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ABSTRACT 



The National Reporting System (NRS) is a project to develop 
an accountability system for the federally funded adult education program. 

The system includes a set of student measures to allow assessment of the 
impact of adult education instruction, methodologies for collecting the 
measures, reporting forms and procedures, and training and technical 
assistance activities to assist states in collecting the measures. This 
implementation document contains five chapters. Following an overview of the 
system and the document, Chapter 1 presents NRS measures, methods, and 
reporting requirements in greater detail. Chapter 2 presents definitions of 
all NRS measures. Chapter 3 presents the data flow framework for the NRS, 
describing how information flows from the classroom and program on to the 
state and federal levels. Chapter 3 also discusses the responsibilities of 
each agency in the data collection and reporting process. Chapter 4 explains 
NRS methodologies and procedures for collecting each measure, including the 
follow-up survey and data matching methods . Chapter 5 contains 
recommendations for local student record systems to enable NRS reporting and 
provides federal reporting tables. Two appendixes include the following: (1) 

a list of NRS advisory board members, pilot test participants, and NRS staff; 
and (2) sample follow-up survey and model methodologies. (Contains 28 
references.) (KC) 
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Chapter I. History and Overview of the 
National Reporting System 



The National Reporting System (NRS) is a project to develop an accountability system for 
the Federally funded adult education program. This system includes a set of student measures to 
allow assessment of the impact of adult education instruction, methodologies for collecting the 
measures, reporting forms and procedures, and training and technical assistance activities to assist 
states in collecting the measures. 



History of the NRS 

The NRS was born in the 1990s, a decade known for its emphasis on accountability of 
Federal programs. During this time, all publicly funded programs and agencies faced increasing 
pressures to demonstrate that they have met their legislative goals and have an impact on their client 
populations. The requirement to demonstrate program impact was mandated in 1993 through the 
Government Performance and Review Act (GPRA). GPRA required all Federal agencies to develop 
strategic plans to ensure that services were delivered efficiently and in a manner that best suits client 
needs, and to develop indicators of performance to demonstrate their agency’s impact. 

In 1995, the U.S. Congress considered eliminating adult education as a separate delivery 
system by integrating the program into a general system of workforce development. Strong and 
convincing data on the impact of adult education at the state and federal levels were demanded to 
demonstrate its importance as a separate education program. Similar demands were raised at the 
state level. In response to these demands, the state directors of adult education asked the Division of 
Adult Education and Literacy (DAEL) to work toward developing a national system for collecting 
information on adult education student outcomes. 

To meet this request, DAEL devoted its March 1996 national meeting of state directors of 
adult education to developing a framework for program accountability. This framework specified the 
purposes of the adult education program, the essential characteristics of an accountability system and 
identified seven categories of outcome measures. At the March 1997 DAEL national meeting, a 
broad group of adult education stakeholders validated the framework, identified outcome measures 
for a new national reporting system, and discussed possible methodologies for the system. Based on 
these decisions, the NRS was designed and formally began in October 1997. 

The proposed voluntary nature of the NRS changed in August 1998, when the Adult 
Education and Family Literacy Act within the Workforce Investment Act (WIA — P.L. 105-220) 
became law. This Act established accountability requirements, including that states develop 
outcome-based performance standards for adult education programs, as one means of determining 
program effectiveness. The NRS mandate was then expanded to establish the measures and methods 
to conform to the Workforce Investment Act requirements. 
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NRS Project Activities 

The goals of the NRS project were to establish a national accountability system for adult 
education programs by identifying measures for national reporting and their definitions, establishing 
methodologies for data collection, developing software standards for reporting to the U.S. 
Department of Education and developing training materials and activities on NRS requirements and 
procedures. The project was designed to conduct these activities in three phases. 

The first phase, standardization , involved the development of standard measure definitions 
for state and local programs, standard data collection methodologies, and software standards for 
automated data reporting. In the summer of 1998, interim software standards were established, 
methodologies were identified for pilot testing and draft definitions for use in the pilot test were 
distributed to adult education stakeholders. 

The pilot test was the second phase of the project and was designed to have a small number 
of volunteer states and local programs test the draft measure definitions and proposed methodologies 
under realistic conditions. The pilot assessed whether the draft measure definitions worked or 
needed refinement, as well as die costs, burden, and other difficulties in collecting the data using the 
proposed methodologies. The pilot test was completed in January 1999. Measures and 
methodologies were revised based on die pilot test. 

The third phase of die project, training and technical assistance , beginning in the summer of 
1999, will support state and local program implementation of the NRS. The different types of 
assistance will include instructional training packets that will be suitable for states to use in a "train 
the trainer" environment; technology-based materials for state and local staff that explain the NRS 
measures and methods; and individual technical assistance to states to support their implementation 
efforts. 



Throughout die course of die project, an advisory board consisting of state directors of adult 
education, representatives from volunteer provider agencies, directors of local adult education 
programs and experts on accountability systems, has guided the project, meeting three times between 
December 1997 and March 1999. The board made significant substantive contributions to the 
measure definitions and methodologies. Participants in the pilot test also provided advice and 
guidance on measures and mediods. Appendix A lists the NRS advisory board, pilot test participants 
and NRS staff. 



Overview of the NRS Measures and Methods 

This document presents products from the first two phases of the NRS project: the measure 
definitions, methodologies and reporting formats for the NRS, which become effective for the 
program year beginning July 1, 2000. The document also includes an overall framework of NRS 
operation at the local, state and Federal levels. 

NRS Measures 

The requirements ofWJA, consensus among the stakeholders and advisory board members, 
and the need for uniform valid and reliable data were major factors guiding development of NRS 
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measures. Other factors affecting development of the measures included the need to accommodate 
the diversity of the adult education delivery system and the need for compatibility of the definitions 
with related adult education and training programs. 

As a state-administered program, the nature of adult education service delivery varies widely 
across states in its goals, objectives and the resources available to states to collect and report data. It 
is especially important that the definitions for outcome measures be broad enough to accommodate 
these differences, yet concrete and standardized sufficiently to allow the NRS to establish a uniform, 
national database. Similarly, other adult education, employment and training programs with which 
adult education works have systems of accountability and outcome measures. 

To ensure this accommodation to the diverse delivery system and compatibility with related 
systems, NRS staff conducted a thorough review of measure definitions planned or in use currently 
by all states and all Federal employment and training programs. To identify state measures used, for 
example, NRS staff conducted an evaluability assessment of all states in early 1998 and obtained 
copies of measure definitions from states that had their own measures. In addition, NRS staff 
reviewed the existing measure definitions used for DAEI/s Annual Statistical Performance Report 
and measures and definitions currently planned by the Department of Education for Title I of WI A. 

A full listing of the main sources consulted in developing the measures and their definitions is 
provided at the end of this chapter. 

Exhibit 1 lists the NRS measures, which include core measures and secondary measures. 

The core measures apply to all adult education students receiving 12 or more hours of service. There 
are three types of core measures: 

❖ Outcome measures , which include educational gain, entered employment, retained 
employment, receipt of secondary school diploma or GED and placement in 
postsecondary education or training; 

❖ Descriptive measures , including student demographics, reasons for attending and student 
status; and 

❖ Participation measures of contact hours received and enrollment in instructional 
programs for special populations or topics (such as family literacy or workplace literacy). 

Performance standards required by WIA will be set for the core outcome measures and awarding of 
incentive grants will be tied to these performance standards. 

The NRS secondary measures include additional outcome measures related to employment, 
family and community that adult education stakeholders believe are important to understanding and 
evaluating adult education programs. States are not required to report on the secondary measures 
and there are no performance standards tied to them. The optional secondary measures will not be 
used as a basis for incentive grant awards. There are also secondary student status measures that 
define target populations identified in WIA. These measures are provided for states that want to 
report on the services provided to these populations. 
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Exhibit 1 

Summary of NRS Measures and Definitions 



Topic 



Measures 



Categories or Definitions 



Core Outcome Measures 



SCtocsccnai. Gains 






Educational gains 



Educational functioning levels in 
reading, writing, speaking and 
listening and functional areas. 
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Entered employment 



Retained employment 

Receipt of secondary school 
diploma or GED 

Placement in postsecondary 
education or training 






Learners who obtain a job by the 
end of the first quarter after exit 
quarter 

Learners who remain employed in 
the third quarter after program exit 

Learners who obtain a GED, 
secondary school diploma or 
recognized equivalent 

Learners enrolling in a 
postsecondary educational or 
occupational skills program building 
on prior services or training 
received 



Core Descriptive and Participation Measures 
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Ethnicity 
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Gender 

Age 



American Indian or Alaskan Native, 
Asian, Native Hawaiian or Pacific 
Islander, Black or African American 
(non-Hispanic), Hispanic or Latino, 
White (non-Hispanic) 

Male, female 

Date of birth 



Status sna Goals 






£ ^ * * s ^ 
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❖ Labor force status 

❖ Public assistance status 

❖ Rural residency 

❖ Disability status 

❖ Learner main and secondary 
reasons or goals for attending 



:*>>>>>>>>:■:* 
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❖ Employed, not employed, not in 
labor force 

❖ Receiving or not receiving 
assistance 

❖ Rural, not rural 

❖ Disabled, not disabled 

❖ Obtain a job, retain current job, 
improve current job, earn a 
secondary school diploma or GED, 
enter post-secondary education or 
training, improve basic literacy 
skills, improve English language 
skills, citizenship, work-based 
project learner goal, other personal 
goal 
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Exhibit 1 (Continued) 

Summary of NRS Measures and Definitions 



Topic 



Measures 



Categories or Definitions 




m 




Contact hours 
Program enrollment type 



❖ Number of hours of instructional 
activity 

❖ ABE, ESL, ASE, family literacy, 
workplace program, homeless 
program, correctional facilities, 
community corrections programs, 
other institutional program 



Secondary Outcome and Student Status Measures (Optional) 



jEmpioymsm 






❖ Reduction in receipt of public 
assistance 



Sili 



fiiil 



Students whose TANF or 
equivalent public assistance grant 
is reduced or eliminated due to 
employment 



W6rk4iased Projecitearner 



iiiiii 



❖ Met work-based project learner 
goal 






Achieved skills for work-based 
project learner activity (activity of at 
least 12 hours and no more than 30 
hours of instruction to teach 
specific workplace skill) 



<?6mmuntty 
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❖ Achieved citizenship skills 



❖ Voting behavior 



❖ General involvement in community 
activities 



Achieve the skills needed to pass 
the citizenship exam 

Learner registers to vote or votes 
for the first time 

Learner increases involvement in 
community activities 



prnity 






❖ Involvement in children’s education 



IIIIII ♦> Involvement in children’s literacy- 




IIIIII 



related activities 



Learner increases help given for 
children’s school work, contact with 
teachers to discuss education, and 
involvement in children’s school 

Learner increases the amount read 
to children, visits libraries, or 
purchases books or magazines for 
children 






r 



Student Status 



Low income status 



lllll!! * Displaced homemaker 







Single parent status 
Dislocated worker 

Learning disabled adult 



Low income, not low income 

Displaced homemaker, not 
displaced homemaker 

Single parent, not single parent 

Dislocated worker, not dislocated 
worker 

Learning disabled, not learning 
disabled 
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Core Outcome Measures 

The central measures of the NRS are the student outcome measures. While by no means the 
only measures that could be used to evaluate adult education programs, the outcome measures 
selected represent what a broad consensus of adult educators believe are appropriate for providing a 
national picture of the performance of the program. The multi-year process employed by the NRS to 
identify and define the measures included input from state directors of adult education, Federal 
education officials, local education providers, representatives of volunteer literacy organizations and 
experts in performance accountability systems. 

The five NRS core outcome measures were selected to address the requirements for core 
indicators of performance in the Adult Education and Family Literacy Act of the WIA. Exhibit 2 
shows how the measures relate to these requirements and goals for adult education stated in the 
legislation. 



Exhibit 2 

Goals and Core Indicators of the WIA 

Adult Education and Family Literacy Act and NRS Core Outcome Measures 



Goals of Adult Education 
Described in the Adult 
Education and Family 
Literacy Act of WIA 


Core Indicators Required by 
the Adult Education and Family 
Literacy Act of WIA 


National Reporting System 
Core Outcome Measures 


Assist adults to become literate 
and obtain the knowledge and 
skills necessary for employment 
and self-sufficiency. 

Assist parents to obtain the skills 
necessary to be full partners in 
their children’s educational 
development. 


Improvements in literacy skill 
levels in reading, writing and 
speaking the English language, 
numeracy, problem-solving, 
English language acquisition, 
other literacy skills. 


❖ Educational gains (achieve 
skills to advance educational 
functioning level) 


Placement in, retention in, or 
completion of, postsecondary 
education, training, unsubsidized 
employment or career 
advancement. 


❖ Entered employment 

❖ Retained employment 

❖ Placement in postsecondary 
education or training 


Assist adults in the completion of 
secondary school education. 


Receipt of a secondary school 
diploma or its recognized 
equivalent. 


❖ Receipt of a secondary 
school diploma or pass GED 
tests 



Educational gain, a key outcome in the NRS, provides a measure of student literacy gains 
resulting from instruction. This measure applies to all students in the program (except pre-designated 
“work-based project learners,” described below). To determine this measure, local programs assess 
students on intake to determine their educational functioning level . There are four levels for adult 
basic education (ABE), two for adult secondary education (ASE) and six levels of English-as-a 
second language students (ESL), Each level describes a set of skills and competencies that students 
entering at that level can do in the areas of reading, writing, numeracy, speaking, listening, functional 
and workplace areas. Using these descriptors as guidelines, programs determine the appropriate 
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initial level in which to place students using a standardized assessment procedure (a test or 
performance-based assessment). The program decides the skill areas in which to assess the student, 
based on student’s instructional needs and goals. 

After a pre-determined amount of instruction or time period determined by each state, the 
program conducts follow-up assessments of students in the same skill areas and uses the functioning 
level descriptors to determine whether the student has advanced one or more levels or is progressing 
within the same level. The state has discretion to establish the student assessment method used 
within the state, as well as procedures for progress assessment. States may also use additional 
educational levels and skill area descriptors, as long as they are compatible with NRS levels and 
skills. 



The remaining core outcome measures are follow-up measures, reported some time after the 
student leaves the program. However, die follow-up measures apply only to students who enter the 
program with goals related to the measures. For unemployed students who enter the program with a 
goal of obtaining employment, there are two measures: entered employment — whether the student 
obtained a job by the end of the first quarter after leaving; and retained employment — whether the 
student still has the job in the third quarter after exit. This measure also applies to employed students 
who have a goal of improved or retained employment. For students whose goal is to advance to 
further education or training, there is a measure of entry into another such program. For students 
who entered with a goal of obtaining a secondary school diploma or passing the GED tests, there is a 
measure of whedier die student obtained the credential. 

Descriptive and Participation Measures 

The NRS descriptive measures collect student demographics, status in several areas and goals 
for attending. These measures allow for a description and understanding of who attends adult 
education programs and for what reasons. The measures also allow for analyses of die performance 
of specific groups of students attending adult education, such as unemployed students or students 
receiving public assistance. The demographic measures include ethnicity, age, and gender, and status 
measures include employment status and whether die student has a disability or is on public 
assistance. 

For die first time at the national level, the NRS requires collection of student goals — both a 
main and a secondary reason — for attending the program. The goals designated are used to compute 
the proportion of students achieving the follow-up measures. 

There are two participation measures, contact hours and program enrollment type, collected 
for bodi descriptive and analytic purposes. These measures record the amount of instruction students 
receive and the number of students attending in areas such as family literacy and workplace literacy. 



Secondary Measures 

The NRS secondary measures are optional measures of student outcomes and status that 
states are not required to collect and that will not be used as a basis for assessing state performance 
under WIA. The NRS includes these measures, however, since many stakeholders during the 
consensus building process believed these measures are important to the identity of the program and 
the goals and purposes of adult education. 
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The secondary measures are in the areas of employment, community and family. The 
employment measure is whether the student’s public assistance grant was reduced or eliminated due 
to employment. This measure applies only to students receiving public assistance upon entry. 

In the area of community, there are three measures covering citizenship, voting and 
community involvement. For students enrolled in citizenship programs there is a measure of whether 
the student has achieved the skills to pass the citizenship exam. Voting for the first time or 
registering to vote and more involvement in community groups or activities are the remaining 
measures. The family measures include increased involvement in children’s literacy activities and in 
children’s education. 

A measure new to national reporting is completed work-based project learner activity . 

Project learners are students enrolled in a class with 30 hours or less of scheduled instruction that has 
a goal of teaching specific workplace-related literacy skills. On enrollment, the learner and the 
program determine die specific skills to be learned and the method to assess the attainment of the 
skills. The assessment must employ a standardized test or be a performance-based assessment with 
standardized scoring rubrics. Programs do not collect the core outcome measures on students 
designated as project learners and these learners are counted separately. This measure is included 
within the NRS to allow states and programs to serve learners with a short-term learning need 
without having a detrimental effect on performance of the core outcome measures. 

Secondary student status measures of low income status, displaced homemaker and single 
parent status are included, as these groups are specific target populations under WIA. States needing 
to report their services to these populations can use these measures, which are defined identically to 
the U.S. Department of Labor definitions. There is also a secondary status measure to identify 
learning disabled adults to assist programs in reaching these students. 



NRS Methodologies 

To help ensure comparability of measures across states, the NRS has established procedures 
for collecting all of the NRS measures. The NRS has three methodologies for collecting measures: 
direct program reporting, local follow-up survey and data matching. With the direct program 
reporting methodology, local programs collect the information directly from the learner while the 
learner is enrolled and receiving instruction. The information is normally obtained as part of the 
intake process, through student assessment or ongoing throughout the course of instruction. 

Measures collected with this methodology are the demographic, student status and student 
participation measures, as well as the educational gain measure and the secondary measures of 
project learner completion and citizenship skill attainment. 

Two methodologies are offered for collecting the NRS core outcome measures that require 
follow-up: the employment-related measures, receipt of secondary diploma or GED and placement 
in postsecondary education or training. Follow-up methodologies also may be used to collect the 
optional, secondary outcome measures. The local follow-up survey methodology employs a survey 
of learners who left the program during the program year. The local program, state or a third-party 
contractor may conduct the survey as long as it includes students from each local program. To 
conduct this survey, programs must include all of the students in the program with one or more of the 
follow-up goals or draw a statistically valid random sample of learners who had one or more of the 
goals. The procedures for conducting the survey are to be determined by the state, but must follow 
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accepted scientific practice for producing valid results. Students with a goal of obtaining 
employment are to be surveyed at the beginning of the first quarter after leaving the program. 
Retained employment must be collected in the third quarter after exit and the other measures can be 
collected at any time during the year. 

An alternative to the survey for collecting the follow-up measures is a data matching 
methodology. Data matching refers to the procedures whereby agencies serving common clients 
pool their data to identify outcomes unique to each program. Matching is achieved using student 
Social Security numbers and is typically done at the state level. For example, to determine whether 
students obtained employment after leaving the program, the state adult education agency would 
match the Social Security numbers and dates of attendance of students who had obtained 
employment in the state wage record data base for the appropriate calendar quarter. States may use 
either follow-up method, or a combination of the two methods, to collect the NRS follow-up 
measures. 



Summary: What’s New 

The NRS makes significant changes to the prior state reporting requirements, the Annual 
Statistical and Performance Report. The following list summarizes the changes from prior reporting 
as a means of highlighting the new and significant features of the NRS. 

❖ The NRS has core measures that meet the requirements of the WIA and apply to adult 
education students who receive 12 or more hours of service. 

❖ While prior reporting has always had a measure of educational gain using functioning 
level descriptors, the NRS makes four major changes to the prior levels. 

S All descriptors have been substantially revised and expanded. There are new 
descriptor categories: “computation” has been changed to “numeracy” and 
workplace skills and technology skills have been added to the functional level 
descriptions. 

S There is now a single set of reading and writing descriptors for both ABE and ESL at 
beginning and intermediate levels. This approach provides a higher degree of 
consistency on the meaning of the levels across program areas. Descriptors in 
reading and writing vary slightly among the ASE and advanced ESL levels to reflect 
the different set of skills needed for the ASE levels. 

S The NRS also has three new levels: an intermediate level for both ESL and ABE, 
formed by breaking up the old beginning and intermediate levels and another ASE 
level, formed by splitting the old ASE level. 

S The NRS descriptors include additional test benchmarks. 

❖ The entered employment, retained employment, receipt of a secondary school diploma or 
GED and placement in postsecondaiy education or training are follow-up measures 
collected on students after they leave the program. These measures apply only to 
students who designated these outcomes as goals on intake into the program. 
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❖ Learner goals or reasons for attending is a new measure used to identify categories of 
students to which the follow-up measures should apply. A main and a secondary goal 
can be identified for each student. Note that goal attainment itself is not an outcome 
measure in the NRS, although it could be computed from information in the NRS and 
used as an outcome. 

❖ States must use a survey methodology, data matching procedures or a combination of the 
methods to collect the follow-up measures. The follow-up measures must be collected 
from all local programs to enable states to meet WIA requirement for evaluating their 
grantees. 

❖ The NRS includes secondary outcome measures that are optional. No performance 
standards will be tied to these measures and they will not be used to determine state 
eligibility for incentive awards under WIA. The secondary measures include welfare 
grant reduction, community- and family -related measures. Most of the secondary 
measures appeared on the prior Annual Performance Report. 

❖ A new secondary measure, attainment of work-based project learner goal is an optional 
measure states may use to identify learners enrolled in an activity of no more than 30 
scheduled hours and designed to teach workplace-related skills. Project learners must be 
identified at enrollment and evaluated with a standardized procedure (test or 
performance-based) to determine whether the skills have been learned. The core 
outcome measures do not apply to project learners and they are counted separately from 
other learners. 

❖ Other secondary measures include five new status measures: low income status, 
displaced homemaker, single parent status, dislocated worker status and learning disabled 
adult. These measures provide a way for states to report on these WIA target populations 
using standard, national definitions. A new secondary measure of learning disabled adult 
is also included. 

❖ Several measures required for the old Annual Performance Report have been dropped, 
including reason for separation before completion and several student status measures. 
Some of the achievements from the prior report have been made secondary measures. 



Overview of This Document 

The remainder of this document presents NRS measures, methods and reporting requirements 
in greater detail. Chapter II presents definitions of all NRS measures and Chapter III presents the 
data flow framework for the NRS. The framework describes how information flows from the 
classroom and program on to the state and Federal levels. The chapter also discusses the 
responsibilities of each agency in the data collection and reporting process. 

Chapter IV explains NRS methodologies and procedures for collecting each measure. 
Guidance on conducting the follow-up survey and data matching methods is also offered. Chapter V 
presents recommendations for local student record systems to enable NRS reporting and provides 
Federal reporting tables. Appendix A lists NRS advisory based members, pilot test participants and 
NRS stalf. Appendix B offers sample follow-up survey and model methodologies. 
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Program. U.S. Department of Education, Office of Vocational and Adult Education, 
Division of Adult Education and Literacy, Washington, DC. 



Condelli, L. & Padilla, V. (1998). State Reporting Systems for Adult Education : Summary of the 
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Education, Springfield, IL. 
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Workforce Programs. Illinois State Board of Education, Springfield, IL. 
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Chapter II. NRS Definitions: Core and Secondary Measures 



The NRS includes core measures and secondary measures . The core measures are required 
and include outcome and descriptive measures and participation measures that reflect the core 
indicator requirements of the Workforce Investment Act. States must report the required measures 
on all students who receive 12 hours or more of sendee . The Department of Education will use these 
measures to judge program performance, including eligibility for incentive grants. The secondary 
measures include additional, optional outcomes related to employment, family and community that 
adult education stakeholders believe are important to understanding and evaluating adult education 
programs. States are not required to report on the secondary measures and there are no performance 
standards tied to them. The optional secondary measures will not be used as a basis for incentive 
grant awards. There are also secondary measures of student status that include target populations 
identified under WIA. These measures are included for states that want to report on sendees 
provided to these populations. The definitions are identical to the Department of Labor definitions to 
aid in uniform reporting under both Title I and Title II of WIA. 

This chapter presents the definitions for the core and secondary NRS measures. Along with 
the definition, the presentation includes the applicable student population to which the measure 
applies; the data collection procedure to be used by states or local programs to collect the measure; 
and Federal reporting method, which describes how the measure is to be reported and used. Chapter 
IV presents more detail on data collection methodologies. 

Core Measures (Required) 



Outcome Measures 

Core Outcome Measure #1: Educational Gain 

Definition: Learner completes or advances one or more educational functioning levels 
from starting level measured on entry into the program. The following tables provide entry-level 
descriptors for the educational functioning levels. 

Applicable Population: All learners. 

Collection Procedure: At intake, an individual learner’s educational functioning level is 
determined within the functional level descriptors, using a uniform, standardized assessment 
procedure approved by the state. The assessment procedure may be a standardized test or a 
performance assessment with standardized scoring protocols. (Note that learners designated as work- 
based project learners need not be assessed in this way, as explained in Chapter I and in the definition 
in this chapter). 

The functional level descriptors describe what a learner entering that level can do in the areas 
of reading and writing, numeracy, speaking and listening and/or functional or workplace skills. The 
local program need not assess the learner in all areas, but the assessment should be in the areas in 
which instruction will be focused. If the learner is functioning at different levels in the areas, the 
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lowest functioning level should be the basis for initial placement. Test benchmarks are provided for 
each level as examples only and these tests do not need to be used when assessing the learner. 

To determine gain, the learner should be assessed again at least once after a standard, 
instructional period, at the end of the class or at the end of the program year, as determined by state 
policy. If more than one assessment is given, the latest assessment should be used to determine 
completion or advancement. An “advance” or “completion” is recorded if, according to a subsequent 
assessment, the student has entry level skills corresponding to one or more levels higher than the 
incoming level in the areas initially used for placement. The lowest functioning level should again 
be used to make this determination. 

State and local programs may use whatever additional level descriptors desired, as long as 
these descriptors can be matched to the NRS definitions. States and local programs may also use any 
assessment procedures desired, as long as the procedures are standardized for all programs in the 
same way (i.e., used consistently and reliably across programs and produce observable measures). 

To assist in placement, test benchmarks are provided for the levels. Tests included are the 
Comprehensive Adult Student Assessment System (CASAS — Life Skills or Employability); Test of 
Adult Basic Education (TABE, Forms 5-6 and 7-8), total reading and total math scale scores and 
grade equivalents; the Adult Basic Learning Examination (ABLE, Forms E-F), reading and math; the 
Adult Measure of Educational Skills (AMES, Forms 1 and 2), reading, total math and 
communication; Student Performance Levels (SPL) for ESL in both speaking and reading; and oral 
scores of the Basic English Skills Test (BEST) for ESL. These benchmarks are provided only as 
examples of how students functioning at each level would perform on the tests. Their inclusion in no 
way is meant to imply that the tests are equivalent or that they should be used as the basis for 
assessment. In addition, the tests do not necessarily measure the same skills. 

Federal Reporting: Total number of learners who complete a level during the program is 
reported and a rate or percentage of level completion can be computed. The number who continue in 
the program after completing a level, the number who fail to complete a level and leave the program 
and the number who remain in the same level are recorded to obtain a fuller picture of student flow 
and retention. 

Core Outcome Measure #2: Entered Employment 

Definition: Learner obtains a job by the end of the first quarter after the program exit 
quarter . 

Applicable Population: Learners who are not employed at time of entry and who have a 
main or secondary goal of obtaining employment. 

Collection Procedure: At intake, the local program collects the individual learner’s 
employment status and employment goal. Entered employment is measured any time from 
enrollment until the end of the first quarter after program exit quarter by local follow-up survey or by 
data matching procedures (see Chapter IV). Note that a job obtained while the student is enrolled 
can be counted for entered employment but is reported on exit from the program. Employment is 
defined as working in a paid, unsubsidized job, or working 15 hours or more per week in an unpaid 
job on a farm or business operated by a family member or the student. The exit quarter is the quarter 
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when instruction ends, the learner terminates, or has not received instruction for 90 days and is not 
scheduled to receive further instruction. 

Federal Reporting: Total number of learners who enter employment is reported to compute 
a rate or percentage, divide this total by the total relevant population (number of learners in the 
workforce who are unemployed at entry and had a main or secondary goal of obtaining employment). 

Core Outcome Measure #3: Retained Employment 

Definition: Learner remains employed in the third quarter after exit quarter . 

Applicable Population: Learners who, at time of entry were not employed and had a main 
or secondary goal of obtaining employment, and who entered employment by the first quarter after 
exit quarter; and learners who are employed at entry and who have a main or secondary goal of 
improved or retained employment. 

Collection Procedure: For students who obtain a job while enrolled, students who obtain a 
job by the end of the first quarter after the exit quarter, and students who are employed at entry and 
had a goal of improved employment, retained employment is measured in the third quarter after exit 
quarter. Data can be obtained either by local survey or by data matching (see Chapter IV). 

Federal Reporting: Total number of learners who retained employment is reported to 
compute a rate or percentage by dividing this total by the total relevant population (number of 
learners in the workforce who were not employed at entry, had a main or secondary goal of obtaining 
employment and who entered employment; and learners employed at entry with a main or secondary 
goal of improved or retained employment). This measure is not reported for learners exiting in the 
third or fourth quarters of the program year if the local survey is used (see page 48). 

Core Outcome Measure #4: Receipt of a Secondary School Diploma or GED 

Definition: Learner obtains certification of attaining passing scores on the General 
Education Development (GED) tests , or who obtains a diploma , or state recognized equivalent , 
documenting satisfactory completion of secondary studies (high school or adult high school 
diploma ). 

Applicable Population: All learners with a main or secondary goal of passing the GED tests 
or obtaining a secondary school diploma or its recognized equivalent, starting in the program year 
that begins July 1, 2001, this measure is collected only for students who exit during the program 
year. 



Collection Procedure: Passage of GED tests or receipt of a secondary school diploma is 
obtained through data match from GED testing agency or from adult high school upon awarding of 
diploma. The program survey methodology may also be used to obtain the information (see Chapter 
IV). Secondary school diploma or GED test passage attainment is recorded any time up to the 
reporting deadline (December 31). 

Federal Reporting: Total number of learners who obtain GED certification and secondary 
school diplomas and number who had this goal is reported. To compute a rate or percentage of 
attainment, the number of students receiving a secondary school diploma or GED is divided by the 
total number of learners who had a main or secondary goal of secondary credential attainment. 
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Core Outcome Measure #5: Placement in Postsecondary Education or Training 

Definition: Learner enrolls in a postsecondary educational or occupational skills training 
program that does not duplicate other services or training received, regardless of whether the prior 
services or training were completed. 

Applicable Population: All learners with a goal of placement in postsecondary education or 
training. Starting in die program year diat begins July 1, 2001, diis measure is only collected for 
students who exit during the program year. 

Collection Procedure: Information on enrollment into training or postsecondary educational 
program is obtained through local survey or data match from other education and training agency, 
such as job training program, adult postsecondary education programs, and community and four-year 
colleges and universities (see Chapter IV). Placement is recorded any time up to the reporting 
deadline (December 31). 

Federal Reporting: Total number of learners who enter postsecondary education or a 
training program and number who had tiiis goal is reported. To compute a rate of placement, die 
number of students enrolling in postsecondary education or training is divided by total number of 
learners witii a goal of advancing to postsecondary education or training. 

Demographic Measures 

Demograph ic Measure # 1 : Eth n icity 

Definition: Learner’s ethnic category to which the learner self -identifies, appears to 
belong to, or is regarded in the community as belonging. The ethnic categories are: 

❖ American Indian or Alaskan Native — A person having origins in any of the original 
peoples of Nordi America, and who maintains cultural identification through tribal 
affiliation or community recognition. 

❖ Asian — A person having origins in any of die original peoples of the Far East, Southeast 
Asia or the Indian subcontinent. These areas include, for example, China, India, Japan 
and Korea. 

❖ Native Hawaiian or Other Pacific Islander — A person having origins as a native of the 
Hawaiian Islands or the other islands of die Pacific, such as the Philippine Islands and 
Samoa. 

❖ Black or African American — A person having origins in any of the Black racial groups 
of Africa, but not of Hispanic culture or origin. 

❖ Hispanic or Latino — A person of Mexican, Puerto Rican, Cuban, Central or South 
American, or odier Spanish culture or origin, regardless of race. 
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❖ White — A person having origins in any of the original peoples of Europe, North Africa, 
or the Middle East, but not of Hispanic culture or origin. 

Applicable Population: All learners. 

Data Collection: At intake, individual learner’s ethnicity is recorded either through 
observation or by self-report of learner. 

Federal Reporting: Total number of learners by ethnic group is reported. 

Demographic Measure #2: Gender 

Definition : Whether the learner is male and female . 

Applicable Population: All learners. 

Collection Procedure: At intake, individual learner’s gender is recorded, either through 
observation or by self-report of learner. 

Federal Reporting: Total number of learners by gender is reported. 

Demographic Measure #3: Age 

Definition: Learner's date of birth. 

Applicable Population: All learners. 

Collection Procedure: At intake, individual learner reports birthdate. 

Federal Reporting: Learner’s age at entry or current age can be computed. Total number of 
learners by age is reported using the following age categories: 16-18, 19-24, 25-44, 45-59, 60 and 
older. 



Student Status Measures 

Student Status Measure # 1 : Labor Force Status 

Definition: Whether the learner is employed , not employed or not in the labor force , 
according to the following criteria: 

❖ Employed — Learners who work as paid employees, work in their own business or farm, 
or who work 15 hours or more per week as unpaid workers on a farm or in a business 
operated by a member of the family. Also included are learners who are not currently 
working, but who have jobs or businesses from which they are temporarily absent. 

❖ Unemployed — Learners who are not working, but are seeking employment, have made 
specific efforts to find a job and are available for work. 

❖ Not in the labor force — Learners who are not employed and are not seeking 
employment. 
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Applicable Population: All learners. 

Collection Procedure: At intake, individual learner reports employment status. 

Federal Reporting: Total number of learners by category is reported. This measure can be 
used with learner goals to calculate rates or percentages for core employment measures. 

Student Status Measure #2: Public Assistance Status 

Definition: Learner is receiving financial assistance from Federal State or local 
government agencies , including Temporary Assistance for Needy Families (TANF) y food stamps , 
refugee cash assistance f old-age assistance , general assistance and aid to the blind or totally 
disabled \ Social Security benefits f unemployment insurance and employment-funded disability are 
not included under this definition . 

Applicable Population: All learners. 

Collection Procedure: At intake, referring agency or individual learner reports public 
assistance receipt. 

Federal Reporting: Total number of learners receiving assistance is reported. This measure 
can be used to calculate rates or percentages for the secondary measure related to public assistance 
reduction. 

Student Status Measure #3: Disability Status 

Definition: Learner has a record of or is regarded as having , any type of physical or 
mental impairment , including a learning disability f that substantially limits or restricts one or 
more major life activities , including walking, seeing, hearing, speaking, learning, and working. 

Applicable Population: All learners. 

Collection Procedure: At intake, referring agency or individual learner reports disability or 
program assesses disability. 

Federal Reporting: Total number of disabled learners is reported. 

Student Status Measure #4: Rural Residency Status 

Definition: Learner who resides in a place with a population less than 2,500 and outside 
an urbanized area . An urbanized area includes a population of 50,000 or more in a city and 
adjacent areas of h igh density . 

Applicable Population: All learners. 

Collection Procedure: At in take, learner residency in a rural area is determined by learner 
self-report or documentation. 

Federal Reporting: Total number of learners living in rural areas is reported. 
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Student Status Measure #5: Learner Reasons or Goals for Attending 

Definition: Learner f s self-identified main and secondary reasons for attending the class or 
program . 

❖ Obtain a job — Obtain full- or part-time paid employment. 

❖ Retain current job— Upgrade skills to enable retention of current job. 

v Improve current job — Obtain a job that has increased pay or benefits, higher level of 
responsibility, or that requires a higher level of skill, compared to current or most recent 
job. 

❖ Earn a secondary school diploma or achieve a GED certificate — achieve sufficient 
skills and credit hours to earn a state accredited secondary diploma or pass the tests of 
General Educational Development. 

❖ Enter postsecondary education or job training— achieve skills to enable enrollment in 
a postsecondary education programs or job training program. 

❖ Improve basic literacy skills — improve overall basic literacy skills. 

♦♦♦ Improve English skills — improve overall skills in the English language (e.g., speaking, 
reading, writing). 

❖ Obtain citizenship skills — Obtain skills to pass the citizenship test. 

❖ Achieve work-based project learner goal— Obtain the skills needed to complete a 
project learner activity (i.e., a course of 12-30 hours duration designed to teach specific 
workplace skills). 

❖ Other personal goal — Any other goal related to instruction with a clearly definable 
outcome (such as passing a driver's test or improved reading ability). 

Applicable Population: All learners. 

Collection Procedure: At intake, learner identifies one main, and as appropriate, one 
secondary reason for attending the program or class within the above categories. The reasons should 
reflect how the learner is to be counted for core outcome measures. 

Federal Reporting: Total number of learners for each type of goal is reported. The number 
of learners in each category is to be used as the denominator to calculate rates or percentages for the 
employment, postsecondary education or training and credential attainment measures. 



Student Participation Measures 

Student Participation Measure #1: Contact Hours 
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Definition: Hours of instruction or instructional activity the learner receives from the 
program . Instructional activity includes any program-sponsored activity designed to promote 
student learning in the program curriculum such as classroom instruction , assessment , tutoring or 
participation in a learning lab . 

Applicable Population: All learners. 

Collection Procedure: Instructor or program staff records contact hours as they are 
provided. Note time spent on assessment can be counted only if the assessment is designed to inform 
placement decisions, assess progress or inform instruction. Time used to take to GED tests for 
example, should not be counted as instructional activity. 

Federal Reporting: Total number of hours is reported. 

Student Participation Measure #2: Program Enrollment Type 

Definition: Learner is enrolled in the following programs or institutions: 

❖ Adult Basic Education Program— A program of instruction designed for adults who 
lack competence in reading, writing , speaking, problem solving or computation at a level 
necessary to function in society, on a job or in the family. 

❖ Adult Secondary Education Program— A program of instruction designed for adults 
who have some literacy skills and can function in everyday life, but are not proficient or 
do not have a certificate of graduation or its equivalent from a secondary school. 

❖ English-as-a-second Language Program— A program of instruction designed to help 
adults who are limited English proficient achieve competence in the English language. 

❖ Family Literacy Programs — A program with a literacy component for parents and 
children or other intergenerational literacy components. 

❖ Workplace Literacy Programs — A program designed to improve the productivity of the 
workforce through improvement of literacy skills needed in the workplace by: 

S providing adult literacy and other basic skills services and activities, including basic 
computer literacy skills; 

S providing adult secondary education services and activities that may lead to the 
completion of a high school diploma or its equivalent; or 

S meeting the literacy needs of adults with limited English proficiency. 

❖ Program for the Homeless — A program designed for homeless adults. Homeless adults 
are adults lacking a fixed, regular nighttime residence or have a residence which is: (1) a 
publicly supervised or privately operated shelter designed to provide temporary living 
accommodations (including welfare hotels, congregate shelters and transitional housing 
for the mentally ill); (2) an institution that provides temporary residence for individuals 
intended to be institutionalized; or (3) a public or private place not designed for, or 
ordinarily used as, a regular sleeping accommodation for human beings. The term 
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“homeless adult” does not apply to any individual imprisoned or otherwise detained 
pursuant to an Act of the Congress or a State law. 

❖ Correctional Facilities — Any prison, jail reformatory, work farm, detention center, or 
any other similar Federal, State, or local institution designed for the confinement or 
rehabilitation of criminal offenders. 

❖ Community Corrections Programs — A community-based rehabilitation facility or 
halfway house. 

❖ Other Institutional Program — Any other medical or special institution. 

Applicable Population: All learners. 

Collection Procedure: At intake, program counts learner as entering the appropriate 
program or category. 

Federal Reporting: Total number of learners in each program or category is reported. The 
number of learners in each program type can be used to analyze the performance of these participants 
separately from the overall adult education population. 

Secondary Measures (Optional) 

Outcome Measures 

Work-based Project Learners 

Project Learner Measure #1: Completed Work-based Project Learner Activity 

Definition: Learner acquires the skills taught in a short-term learning course designed to 
teach specific work-based skills. A short-term course is an instructional program of at least 12 
hours but no more than 30 hours duration. 

Applicable Population: Learners enrolled in a short-term course and designated at entry as 
work-based project learners. 

Collection Procedure: At intake, the individual learner is identified as a work-based project 
learner and as entering a project learning instructional activity. The instruction must be designed to 
teach work-based skills and must specify the educational outcomes and standards for achievement. 
The final assessment of skill achievement must be a standardized test or a performance-based 
assessment with standardized scoring rubrics. 

Federal Reporting: Total number of learners who completed a work-based project learner 
activity is recorded. A rate or percentage can be computed by dividing this total by the total relevant 
population (number of work-based project learners). Project learners are not counted for the 
educational gain measure and are not assigned an educational functioning level. No core outcome 
measures are reported for project learners. 
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Employment Measures 

Secondary Employment Measure # 1 : Reduction in Receipt of Public Assistance 

Definition: Learner’s Temporary Assistance to Needy Families (TANF) grant , or 
equivalent public assistance grant, is reduced or eliminated due to employment or increased 
income . 



Applicable Population: Learners who are receiving TANF or equivalent public assistance 
grant at the time of enrollment in the program. 

Collection Procedure: At intake, individual learner’s public assistance receipt status (TANF 
or equivalent) is collected. Information on reduction or elimination of the grant is recorded by local 
survey or data matching methodology (see Chapter IV). 

Federal Reporting: Total number of learners whose grant is reduced or eliminated is 
reported and a rate or percentage can be computed by dividing this total by the total relevant 
population (number of learners on public assistance on entry into the program). Grant reduction may 
be reported at any time during the program year. 

Community Measures 

Secondary Community Measure #/; Achieved Citizenship Skills 
Definition: Learner attains the skills needed to pass the U.S. citizenship exam . 

Applicable Population: All learners with a main or secondary goal of obtaining citizenship 

skills. 



Collection Procedure: Attainment of skills needed to pass the U.S. citizenship exam are 
assessed and recorded by program staff. 

Federal Reporting: Total number of learners who obtained skills to pass the citizenship 
exam is reported. A proportion or rate can be computed by dividing this total by the total relevant 
population (number of learners who enrolled in citizenship classes or had a main or secondary goal 
of citizenship). 

Secondary Community Measure #2: Voting Behavior 

Definition: Learner registers to vote or vote for the first time during instruction anytime 
during the program year . 

Applicable Population: All learners who, at time of enrollment, are not registered to vote or 
who have never voted. 

Collection Procedure: Information on voting or voter registration is recorded by local 
program survey (see Chapter IV) or direct program report. 



Chapter II, NRS Definitions: Core and Secondary Measures 



27 



Measures and Methods for the National Reporting System for Adult Education 



Federal Reporting: Total number of learners who registered to vote or voted for the first 
time is reported. 

Secondary Community Measure #5: General Involvement in Community Activities 
Definition: Learner increases involvement in the following community activities: 

❖ Attending or organizing meetings of neighborhood, community or political organizations; 

❖ Volunteering to work for such organizations; 

❖ Contributing to the support of such organizations; and 

❖ Volunteering to work on community improvement activities. 

Applicable Population: All learners. 

Collection Procedure: Information on increased involvement in community activities is 
recorded through local program survey (see Chapter IV) or direct program report. 

Federal Reporting: Total number of learners who increase community involvement in any 
measure is reported. 

Family Measures 

Secondary Family Measure #7: Involvement in Children ’s Education 

Definition : Learner increases involvement in the education of dependent children under 
his or her care, including: 

❖ Helping children more frequently with their school work; 

❖ Increased contact with children’s teachers to discuss children’s education; and 

❖ More involvement in children’s school, such as by attending school activities and parent 
meetings, and volunteering to work on school projects. 

Applicable Population: All learners enrolled in programs that include a focus on family 

literacy. 

Collection Procedure: Information on increased involvement in children’s education is 
recorded through local program survey (see Chapter IV) or direct program report. 

Federal Reporting: Total number of learners who increase involvement in any area is 
reported. A rate or percentage can be computed by dividing this total by the total relevant population 
(number of learners in programs that include a family literacy focus). 

Secondary Family Measure #2: Involvement in Children ’s Literacy- Related Activities 
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Definition: Learner increases involvement in the literacy-related activities of dependent 
children under his or her care, including: 

❖ Reading to children; 

❖ Visiting a library; and 

❖ Purchasing books or magazines for children. 

Applicable Population: All learners enrolled in programs that include a focus on family 

literacy. 

Collection Procedure: Information on increased involvement in children’s literacy related- 
activities is recorded through local program survey (see Chapter IV) or direct program report. 

Federal Reporting: Total number of learners who increase involvement in any area is 
reported. A rate or percentage can be computed by dividing this total by the total relevant population 
(number of learners in programs that include a family literacy focus). 

Student Status Measures 

Secondary Student Status Measure # 1 : Low Income Status 

Definition: The learner receives or is a member of a family that received a total family 
income in the six months prior to enrollment of 70 percent of the income level standard for a 
family of that size; or the learner is receiving or is a member of a family that is receiving cash 
assistance payments from Federal or state agencies or Food Stamps; or the learner can be 
designated as homeless under the McKinney Act . 

Applicable Population: All learners. 

Collection Procedure: At intake, referring agency or individual learner reports whether the 
learner is low income. 

Federal Reporting: Total number of low income learners is reported. 

Secondary Student Status Measure #2: Displaced Homemaker 

Definition: Learner has been providing unpaid services to family members in the home 
and has been dependent on the income of another family member but is no longer supported by 
that income; and is unemployed or underemployed and is experiencing difficulty in obtaining or 
upgrading employment. 

Applicable Population: All learners. 

Collection Procedure: At intake, referring agency or individual learner reports whether the 
learner is a displaced homemaker. 

Federal Reporting: Total number of displaced homemakers is reported. 
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Secondary Student Status Measure # 3 : Single Parent Status 

Definition: Learner has sole custodial support of one or more dependent children. 

Applicable Population: All learners. 

Collection Procedure: At intake, referring agency or individual learner reports whedier the 
learner is a single parent. 

, Federal Reporting: Total number of single parents is reported. 

Secondary Student Status Measure #4: Dislocated Worker 

Definition: An individual who received an individual notice of pending or actual layoff 
from a job, or an individual who received a publicly announced notice of pending or actual layoff. 

Applicable Population: All learners. 

Collection Procedure: At intake, referring agency or individual learner reports whether he 
learner is a displaced worker. 

Federal Reporting: Total number of dislocated workers is reported. 

Secondary Student Status Measure #5: Learning Disabled Adult 

Definition: Learners with IQs in the low-average and above (70+ to any level) who have 
deficits (related to neurological impairments) in capacity in defined limited areas related to 
learning, including dyslexia (reading disability), dysgraphia (writing disability), dyscalculia (math 
disability), and who have a history of previous educational efforts. 

Applicable Population: All learners. 

Collection Procedure: At intake, referring agency or individual learner reports whether the 
learner is learning disabled. 

Federal Reporting: Total number of learning disabled adults is reported. 
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Chapter III. The NRS Data Flow Framework 



The goal of the NRS is to produce a set of measures that describe adult education students 
and the outcomes they achieve due to their participation. These measures will be used at the state 
and national levels to demonstrate whom the adult education program serves and its impact on 
learners’ educational and employment-related outcomes. States and local programs can also use the 
data for program management and improvement activities and to develop performance standards now 
required by the WIA. This chapter presents the NRS framework for data flow from local programs 
to the U.S. Department of Education. It also summarizes the roles and responsibilities of local 
programs, the states and the Federal government in the operation and maintenance of the NRS. 



Data Flow Framework of the NRS 

The development of a national database for adult education requires the close collaboration 
among the Federal Department of Education, state education agencies, and local programs, Each 
entity has an essential role in the operation and maintenance of the system that helps ensure the 
collection of valid and reliable data from programs and across the states. These roles within the NRS 
are summarized below. 

❖ The Federal role is to establish the NRS measures, methods and reporting requirements, 
to ensure valid and reliable data, provide assistance to states in understanding and 
implementing these requirements, monitor the system to ensure it is producing valid and 
reliable measures, report the data to federal agencies and decide on state incentive awards 
based on NRS data. 

❖ States are responsible for implementing NRS measures, methods and requirements in a 
way that meets Federal guidelines; for providing resources, training and support for data 
collection to local programs; monitoring local programs; and for funding programs. 

States are to use NRS measures to promote continuous improvement based in part on 
their performance on NRS measures. 

❖ Local programs are responsible for allocating sufficient resources to collect the NRS 
measures and reporting them according to state requirements. 

Exhibit 3 shows the general data flow framework envisioned for the NRS by following the 
movement of data at each of these three levels. At the local program level, each of the program’s 
instructional sites collects measures from students at three time periods. Upon intake into the 
program, local staff collects descriptive measures — demographic information, student goals and 
status measures — and conducts an assessment of the student’s educational functioning level for 
placement. During the course of instruction, program staff (typically the teacher) provides at least 
two additional measures about the student: contact hours or attendance and a progress assessment. 

The progress assessment is usually administered at a time established by the local program or state 
policy and may be at the end of the course of instruction, at the end of the program year or after a set 
number of instructional hours. Other student outcomes attained may also be recorded on the update. 
When this update information is collected is also determined by local or state policy, and may be on 
an ad hoc basis, or at set times during the year, such as monthly or quarterly. 
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In most states, local programs will also be required to collect the core follow-up measures on 
students. These measures include the employment-related measures, measures on placement in 
postsecondary education or training and whether the student received a GED or other secondary 
credential. NRS follow-up procedures require program staff to collect the employment measures 
through a student survey in the first and third quarters after students’ exit quarters, or by matching 
procedures. In states that use the data matching methodology (described in Chapter IV), collecting of 
follow-up data becomes a state responsibility. 

Local programs must combine all of the measures collected at each instructional site into an 
individual student record system. This type of system is essential to the NRS, since it allows local 
programs to conduct the types of analysis of outcomes for specific student groups for reporting and 
program management. For example, only an individual record system would allow analysis of such 
issues as whether a specific type of student (such as students with employment-related goals) 
achieved their outcomes, or the number of instructional hours needed by groups of students to 
advance an educational level. The NRS does not specify the software or design of the student record 
system, but leaves this to local and state discretion. 

Exhibit 3 also shows the movement of NRS data from the local program to the state level. 

Each local program must submit their data to the state education agency to enable the state to develop 
a statewide adult education database. The state data submission is often on an annual basis, although 
some states require more frequent submission. This state database may contain aggregated data from 
local programs or it may consist of the individual student records from programs. At the end of the 
program year, states must submit data in aggregated data tables to the U.S. Department of Education, 
which maintains a national database. This submission is required to be in electronic form, either 
using software developed for this purpose by the Department by using state-developed software. 

Local Role 

Exhibit 3 emphasizes the role programs, states and the Federal government have in 
supporting the NRS. Local programs are on the front lines of the system, as they must allocate 
sufficient resources to collect the information from students — the descriptive, participation and 
outcome measures that comprise the database. For these data to be meaningful on a statewide and 
national basis, however, data collection procedures must be standardized across all programs within 
each state. That is, the data must be defined and collected in the same way by all programs to make 
it comparable. Due to this central role of local programs, local staff needs ongoing training and 
assistance to achieve this standardization in the following areas: 

❖ Understanding the definitions of each measure and clear guidelines on how to record 
these measures, including how to handle missing or incomplete data; 

v Understanding of, and compliance with, the state-defined procedures for assessing 
students for placement into educational functioning levels and assessing progress; 

❖ Following procedures for implementing the follow-up survey, if it is conducted by the 
program; and 

❖ Understanding how to correctly record and report data to the state. 

It is the state responsibility to provide training and technical assistance to local programs on these 
areas. 
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State Role 

The NRS gives the state the primary responsibility for implementing the procedures most 
applicable to its environment and delivery system and ensuring that local programs follow these 
uniform data collection policies and procedures. States will need to achieve this uniformity through: 

Ongoing staff development of local staff; 

'•* Technical assistance to local programs where necessary; and 

❖ Implementing quality control procedures to ensure that local data procedures are accurate 
and complete. 

In addition, the state must establish an individual student record system employing a relational 
database for local programs. The design and operation of this system is left to state discretion, as the 
NRS requires only reporting of aggregated data tables at the Federal level. States are also required to 
use NRS data to promote continuous improvement of local programs. 

A critical element to the NRS is the ability for states and local programs to measure and 
document learning gains on the educational functioning levels. Due to the variation in each state’s 
instructional emphasis, goals and assessment procedures, however, the NRS allows states to establish 
their own procedures for student placement and assessment to measure learning gains. 

Consequently, key state responsibilities are to establish statewide uniform methods of student 
assessment, provide staff development to local staff on these methods, monitor local programs to 
ensure methods are followed and provide technical assistance on the methods where needed. The 
assessment may be a standardized test or performance-based assessment with standardized scoring 
rubrics. Chapter IV provides further information on this issue. 

Federal Role 

At the Federal level, the Department of Education’s role has been to establish the NRS 
procedures through an inclusive process that has not only addressed Federal legislation, but has been 
responsive to state and local concerns and coordinated with Federal partner agencies. The 
methodology and definitions have been developed and pilot tested and reporting guidelines have 
been developed. The Department of Education plans to monitor implementation of the NRS, conduct 
quality control of state procedures and provide ongoing technical assistance and training to states. 

This training will be designed to support state efforts to train staff and implement the NRS to produce 
valid, uniform data. Learning to use data more effectively for program improvement and 
accountability will also be a focus of training as a means to enhance the value of the NRS and 
encourage adoption and support of the system. Technical assistance materials to states on issues 
such as local program quality control and assessment procedures will also be provided. 



Federal and State Accountability Systems 

Tlie NRS includes a set of measures identified by stakeholder groups as central for assessing 
the performance of the adult education program. The NRS also establishes uniform methodologies 
for these measures to increase data validity by helping to ensure consistency in collecting measures 
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across states. The objective is to create a national database of information and outcomes about the 
program to demonstrate its effectiveness and meet WIA accountability requirements. To promote 
program effectiveness, WIA requires states to set performance standards on the core indicators and 
ties incentive grants to these standards. 

WIA also requires states to evaluate local program performance on these same indicators as 
one condition of local funding. States may also use any other indicators of their choosing in 
evaluating programs and making funding decisions. The NRS core measures, collected for national 
purposes, do not limit or preclude use of other measures. Indeed, the inclusion of the secondary 
measures in the NRS framework is intended to provide states with additional options on such 
measures. For example, states that wish to place a greater emphasis on family literacy or community 
involvement could include the secondary measures in these areas in their evaluation of local 
programs and fund them according to performance on these measures. Similarly, states wanting to 
emphasize serving students on public assistance could use the measure of welfare reduction in local 
performance evaluations. 

Nothing in the NRS framework is intended to exclude the use of additional measures in state 
and local evaluation and funding decisions. While some states may wish to design their entire 
evaluation systems solely around NRS measures, the NRS measures in other states may be part of a 
broader set of measures and factors used for performance evaluation and accountability. 



Summary of the NRS Data Flow Framework 

The NRS is designed to develop a national database describing adult education students and 
their outcomes. The data include descriptive measures, such as student demographics and goals; 
attendance hours; and student outcomes, including learning gains, attainment of GED or secondary 
school credentials, placement in postsecondary education or training and employment-related 
measures. 

Local programs have primary responsibility for collecting these measures using valid, 
uniform procedures to ensure comparability across programs and must maintain these data in an 
individual student record system. To ensure that educational gains are standardized, programs must 
have standardized methods for assessing students at intake and following instruction. In most states, 
programs must also conduct a follow-up survey on students. 

States must ensure that measures of educational gains are meaningful by establishing a 
standardized assessment system, based on tests or authentic performance. States are also responsible 
for providing staff development and technical assistance to local programs to ensure measures and 
methods are collected in accordance with NRS guidelines and implement quality control procedures 
to ensure data validity. In addition, the state must maintain a state database that includes data from 
all programs. States are also responsible for data matching procedures when they employ that 
follow-up methodology. 

At the Federal level, the Department of Education has supported the establishment of a 
national database for adult education by developing the framework and measures for the NRS. 
Federal assistance will continue through training and technical assistance, development of materials 
for state staff and by quality control of state data collection efforts. The Department of Education 
will also maintain the national database of measures. 
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What the NRS Is Not 

While tlie foregoing section has explained the chief characteristics of the NRS — what the 
system is — one way to clarify it further is to describe what the NRS is not. 

❖ The NRS does not preclude collecting other measures or coding categories. States 
and local programs may collect and report on any additional measures desired, as long as 
the required NRS core measures are also collected. Additional coding categories for 
measures (such as for ethnicity) may also be collected, as long as the additional coding 
categories can be combined into NRS coding categories. 

❖ The NRS does not require a specific type of record system or software. Local 
programs may use any software, as long as it includes the NRS measures and coding 
categories, and can produce tire Federal reporting tables. 

❖ The NRS does not specify a specific assessment or test. States and local programs may 
use whatever assessment procedures or tests are appropriate for their instructional 
approach, as long as both initial and subsequent assessments are administered in a 
standardized way throughout the state. Further, a state or program may use an authentic 
or other performance-based assessment, as long as it has a standardized scoring rubric 
that teachers or assessors are trained to use. 

❖ The NRS does not preclude using additional educational functioning levels. To 

measure educational learning gains, the NRS requires programs to place students in one 
of six educational functioning levels according to their abilities in designated skill areas 
(see Chapter II). Programs may use additional educational levels, as long as these 
additional levels can be directly related to the NRS levels. 

❖ The NRS does not specify the skill areas to teach or assess students. To measure 
learning gains, the NRS describes educational functioning levels in which to place 
students and assess their progress. The levels describe skills in reading, writing, speaking 
and listening, numeracy, functional and workplace areas. Local programs, however, 
determine which areas to assess students and on which to focus instruction. 
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Chapter IV. NRS Methodologies 



One of the most significant changes the NRS makes over existing reporting procedures is the 
establishment of methodologies for collecting data. Standardized procedures, along with definitions 
of the measures, are essential to promoting validity and reliability of data and ensuring that the 
measures have the same meaning across programs and states. 

The NRS has three methodologies for collecting measures: direct program reporting, local 
follow-up survey and data matching. With the direct program reporting methodology, local 
programs collect the information directly from the learner while the learner is enrolled and receiving 
instruction. The information is normally obtained as part of the intake process, through student 
assessment or is ongoing throughout the course of instruction. Measures collected with this 
methodology are the demographic, student status and student participation measures, as well as the 
educational gain measure and secondary measures of project learner completion, citizenship skill 
attainment, and other optional measures. 

The local follow-up survey methodology employs a universe or random sample survey of 
learners after they exit the program. The local program, state or a third-party contractor may conduct 
die survey as long as the sample includes students from each local program. Data matching 
methodology refers to die procedures whereby agencies serving common clients pool their data to 
identify outcomes unique to each program. Matching is achieved using Social Security numbers and 
is typically done at die state level. For example, wage records may be matched for adult education 
students who had a goal of obtaining employment to determine whether they got a job. 

Both the follow-up survey and data matching methods collect follow-up measures required 
for the NRS — the employment-related measures, obtaining a secondary diploma or passing the GED 
and advancing to further education and training. Follow-up methodologies may also be used to 
collect the optional, secondary outcome measures. States may use either follow-up method, or a 
combination of the two methods, to collect the NRS follow-up measures. Exhibit 4 summarizes the 
methodologies for each NRS measure. 

The first part of this chapter provides guidelines and explains the coding categories for the 
direct program report methodology. The second part of this chapter offers guidelines for conducting 
the local follow-up survey and a discussion of data matching models. The chapter concludes with a 
brief section on promoting data quality. Appendix B provides a sample follow-up survey and a 
model procedure for conducting a survey. 
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Exhibit 4 

NRS Measures and Methodologies 



| Core Measures (Required) 


Methodology 


Outcome Measures 


tducational Gain 


Direct program report (assessment) 


Entered Employment 


Local follow-up survey or data matching* 


Retained Employment 


Local follow-up survey or data matching 


Receipt of a Secondary School Diploma or GED 


Local follow-up survey or data matchinq 


Placement in Postsecondary Education or Training 


Local follow-up survey or data matching 


Demographic Measures j 


Ethnicity 


Direct program report (intake) 


Gender 


Direct program report (intake) 


Age 


Direct program report (intake) 


Student Status Measures 


Labor force status 


Direct program report (intake) 


Public assistance status 


Direct program report (intake) 


Disability status 


Direct program report (intake) 


Rural residency status 


Direct program report (intake) 


Reasons for attending (main and secondary) 


Direct program report (intake) 


Student Participation Measures 




Contact hours | 


^ Direct program report (ongoing during instruction) 


Program enrollment type 


Direct program report (intake) 


Secondary Measures (Optional) 


Work-based project learner achievement 


Direct program report (assessment) 


Reduction in public assistance 


Local follow-up survey or data matchinq 


Achieved citizenship skills 


Direct program report (assessment) 


Voting behavior 


Local follow-up survey or direct program report 


Involvement in community activities 


Local follow-up survey or direct program report 


Involvement in children’s education 


Local follow-up survey or direct proqram report 


Involvement in children’s literacy activities 


Local follow-up survey or direct program report 


Low income status 


Direct program report (intake) 


Displaced homemaker 


Direct program report (intake) 


Single parent status 


Direct program report (intake) 


Dislocated worker status 


Direct program report (intake) 


Learning disabled adult j 


Direct program report (intake) 



'Direct program reporting is possible if learner obtains job while enrolled. 



Direct Program Report Methodology 

Procedures for Measuring Educational Gains 

The educational gain measure is probably the most important single measure in the NRS. All 
participants involved in the development of the NRS agreed that adult education is primarily an 
educational program designed to teach literacy skills. Literacy gains are also explicitly identified in 
the Workforce Investment Act. Besides being a critical indicator of how well adult education 
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programs are doing in fulfilling their primary mandate, how learning gains are defined directly 
affects instructional design, program content, curriculum and assessment. 



Educational Functioning Levels 



The NRS approach towards measuring educational gain is to define a set of educational 
functioning levels in which students are initially placed, based on their ability to perfonn literacy- 
related tasks in specific content areas. After a set time period or number of instructional hours, 
students are again assessed to determine their skill level. If their skills have improved sufficiently to 
be placed one or more levels higher, an “advance” is recorded for that student. Chapter II provides 
the educational functioning level descriptors. 

The NRS divides educational functioning into six levels for both ABE and ESL. The levels 
for ABE are beginning literacy, beginning basic education, low and high intermediate basic 
education, and low and high adult secondaiy education. Each ABE level describes the skills in basic 
reading and writing, numeracy, and functional and workplace skills that a person functioning at that 
level could be expected to perform. The six ESL levels are beginning literacy, beginning ESL, low 
and high intermediate ESL, low advanced ESL and high advanced ESL. The ESL levels describe 
speaking and listening skills, as well as basic reading and writing skills, and functional workplace 
skills that a person functioning at that level can be expected to perform. States and local programs 
may use additional or different skills and levels if desired, as long as these skills and levels can be 
matched to the NRS levels and skills. 

At the low and intermediate levels, the basic reading and writing skills are identical for both 
ABE and ESL. At the higher levels (secondary level for ABE, advanced levels for ESL), the reading 
and writing skills are designed to be slightly higher for ABE than for ESL, since the adult secondary 
level is designed to be the highest level. The functional and workplace skills for ABE and ESL also 
differ somewhat by having a stronger second language focus for ESL. Speaking and listening skills 
are only described for ESL, and numeracy for ABE to reflect common instructional practice. 
However, programs may apply the numeracy descriptors to ESL students and the speaking and 
listening descriptors to ABE students, if the needs of the student and the program’s instructional 
approach warrant this approach. 

Local programs should use the following guidelines to place students in levels. 

❖ The descriptors describe what a typical student functioning at that level should be able to 
do, and are thus entry) level descriptors. When the student has the skills at one or more 
levels above the placement level, he or she can be considered as completing that level and 
can advance to the next level. 

* Students do not need to be assessed in all of the areas described in the level descriptors. 
The local program must decide, in accordance with state guidelines, the skill areas most 
relevant to the student’s needs or the program’s curriculum, and assess students in these 
areas. For example, if a student’s goal is to improve reading skills, the reading and 
writing descriptors would serve as the primary guide for placement; or if the program has 
a strong life skills emphasis, the functional and workplace descriptors would guide 
placement. 
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* If multiple skill areas are assessed and the student has different abilities in different areas, 
the program should place the student according to the lowest functioning level. For 
example, if a student was at the beginning level in reading and the low intermediate level 
in numeracy, the student would be placed in the beginning level. The lowest functioning 
level should also be used to determine educational gain in subsequent assessments. 



Assessment Guidelines 

To ensure comparability of the meaning of the educational functioning levels across all 
programs in the state, all programs must use a standardized assessment procedure approved by the 
state when determining students educational functioning levels. The assessment procedure may be a 
standardized test, or a performance-based assessment with a standardized scoring rubric. If 
performance-based assessment is used, the scoring rubrics should be based on objective, observable 
criteria and program staff should be trained in scoring to ensure that the measures are valid and 
reliable across programs and students. 

To assist in placement decisions, test benchmarks are provided for the levels. Tests included 
are the Comprehensive Adult Student Assessment System (CASAS— Life Skills or Employability); 
Test of Adult Basic Education (TABE, Forms 5-6 and 7-8), reading and total math scale scores and 
grade equivalents, the Adult Basic Learning Examination (ABLE Form E-F), reading and math; the 
Adult Measure of Essential Skills (AMES, Forms 1-2 for ABE), reading, computation and 
communication; Student Performance Levels (SPL) for ESL in speaking, reading, and writing; and 
scores on the oral Basic English Skills Test (BEST). These benchmarks are provided as examples of 
how students functioning at each level would perform on the tests. The tests should not be 
considered equivalent, however, and do not necessarily measure the same skills. In addition, diese 
tests are offered only as examples and their inclusion does not imply that these tests must or should 
be used in the detenu ination of educational functioning levels. 

The NRS requires that local programs assess and place all students into an educational 
functioning level at intake and at least one other time during the program year. Programs should 
administer die initial assessment at intake or within a short period thereafter and administer follow- 
up or post-test assessments according to state policy. The follow-up assessment should be after a set 
instruction time, either in hours (e.g., after 50 hours of instruction) or months (e.g., die last two 
weeks of November, last week of instruction). For the purpose of NRS reporting, the program 
should use a different form of the same test for the follow-up assessment as for the initial assessment, 
or if a performance-based assessment is used, the student’s performance should be assessed in the 
same area as used for placement. For example, if a student was placed based on performance on a 
reading task, an equivalent reading task should be used to assess progress in subsequent assessments. 

If more than one assessment is administered, the program should use the last assessment 
given as the basis for determination of the educational level. If no follow-up assessment of the 
student is made during the program year, the student must be counted as remaining in the same 
educational level as upon entry and cannot be reported as advancing to a higher functioning level. 



State Responsibilities in Assessment 

The NRS gives the state substantial freedom to determine the assessment policy and 
procedures used by local programs. States and local programs decide the skill areas in which to 
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assess students, the tests and assessment procedures to use and when to conduct follow-up 
assessments. This flexibility in assessment permits states and local programs to tailor their 
instructional activities and curriculum to best meet their priorities and the needs of individual 
students. However, for the educational functioning levels to be meaningful, the assessment needs to 
be administered in a standardized and consistent way by all programs in each state. If these 
procedures are not followed correctly or consistently, the determination of educational functioning 
level will be invalid or not comparable across programs or possibly even within programs, making 
the data validity questionable. 

In summary, the NRS gives states the freedom to determine assessment policy and 
procedures, as well as responsibility for training local staff on procedures, ensuring that lpcal 
programs follow state policy and that information on student educational levels is reported 
consistently and reliably across the state. To implement assessment procedures that produce this type 
of data, states must do the following. 

❖ Determine the assessments or tests that local programs should follow. These assessments 
may be standardized tests or performance-based assessments that have standardized 
scoring rubrics that are observable and objective. 

❖ Train local staff in the proper use, administration and scoring of the chosen assessments 
and provide technical assistance to local program staff as needed. Training is especially 
important when alternative or performance-based assessment is used to ensure valid and 
reliable scoring. 

❖ Establish a policy for when progress assessments are to be administered to students. 

❖ Conduct periodic quality control reviews of assessment procedures to ensure local staff 
are correctly following procedures. Guidelines on quality control are provided at the end 
of this chapter. 

Assessing Work-based Project Learner Completion 
and Citizenship Skills 

As discussed in Chapters I and II, the NRS establishes a set of secondary outcome measures. 
While these measures are optional, states that elect to report them are encouraged to follow a 
standard methodology to promote national comparability. Two of the measures — completed work- 
based project learner activity and achievement of skills to pass the citizenship examination — require 
student assessment. 

Work-based project learners are enrolled in an instructional or training course that has at least 
12 hours and no more than 30 hours of scheduled instruction. The course must be designed to teach 
work-based literacy skills. The skills the student is to learn, as well as the method for assessing these 
skills and the standards for achievement, must be explicitly stated prior to beginning the course. To 
count as having completed the activity, the learner must demonstrate achievement of the skills at the 
level of the agreed upon standard. As with other student assessments within the NRS, the assessment 
must either be a standardized test or performance-based assessment with standardized scoring 
rubrics. It is the state's responsibility to establish and monitor the assessment process and train staff 
on the use of the assessment procedures. 



Chapter IV. NRS Methodologies 



41 



Measures and Methods for the National Reporting System for Adult Education 



Students designated as work-based project learners should have “achieve work-based project 
learner goal” designated as their primary goal for attending. Once a student is designated as a work- 
based project learner, that student is not assigned an educational functioning level and no additional 
outcomes are to be collected on that learner. The learner is reported on the NRS reporting tables, 
however (see Chapter V). 

Work-based project learning should not be confused with workplace literacy programs, 
which are also designed to teach workplace skills. Workplace literacy programs are of longer 
duration or are open-ended and generally teach a broader range of literacy skills (see definition in 
Chapter II). Students enrolled in workplace literacy programs are counted under the required core 
outcome measures. 

A second optional measure requiring student assessment is achievement of skills to pass the 
citizenship exam. This measure is included to document learning gains of students who are enrolled 
in classes designed to give them the literacy skills and substantive knowledge to pass this exam. 
These students should have obtain citizenship skills designated as their primary goal for attending. 

To determine whether students have achieved these skills, program staff should administer a state 
approved test that measures the relevant skill areas, such as a practice citizenship test, sample forms 
and speaking tests, at the conclusion of the citizenship class. It is the state’s responsibility to ensure 
that programs use an appropriate test and establish the standards for passing this test, as well as to 
train and monitor local staff in its use, if this measure is to be reported. 



Procedures for Collecting Student Demographic and Status Measures 

The NRS measures include demographic, student status and participation measures. Adult 
education programs have routinely collected these measures and consequently, the collection of them 
in the NRS should have little or no impact on existing data collection procedures. These measures 
are typically collected at intake into the program on a form completed either by intake staff or by the 
student directly. Chapter II contains the definitions of these measures and categories. There are also 
three optional student status measures described here. 

❖ Ethnicity — Student ethnicity should be coded into one of the following categories by the 
student or through observation by staff: 

S American Indian or Alaskan Native 

S Asian 

S Native Hawaiian or Pacific Islander 
S Black or African American, not Hispanic Origin 
S Hispanic or Latino 
S White, not Hispanic Origin 

❖ Gender — Student gender should be coded as male or female by the student or through 
observation by staff. 
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❖ Age — Student birthdate should be recorded by the student or by staff from 
documentation, such as a student identification or documentation from a referring 
agency. Age should not be coded into categories (e.g., 16-21, 22-35), as coding in this 
way precludes use of age as an individual variable in data analyses. 

❖ Labor force status — Student employment status at the time of entry into the program 
should be recorded by the student or by staff from documentation from a referring agency 
into the following categories: 

S Employed 

S Unemployed 

S Not in the labor force 

❖ Public assistance status — Record whether the student is receiving any of the following 
forms of financial assistance from public agencies. Public assistance status should be 
recorded by the student or by staff from documentation from a referring agency into the 
following categories: 

S Temporary Assistance to Needy Families (TANF) or equivalent general assistance 
S Food stamps 
S Refugee cash assistance 
S Old-age assistance 
S Aid to the blind or disabled 

❖ Disability status — Record whether the student has a physical, mental or learning 
disability that restricts one or more major life activities, such as walking, seeing, hearing 
or learning. A disability should be recorded if it can be directly observed, is documented, 
or can be assessed through a valid assessment instrument or procedure designed to 
identify disabilities. 

❖ Rural residency status — Record whether the learner resides in a rural area — i.e., a place 
with a population of less than 2,500 and not near any metropolitan area with a population 
greater than 50,000 — from student self-report or documentation. 

❖ Learner goals or reasons for attending — The student can designate at intake a main 
reason and a secondary reason for attending the adult education class. At least one goal 
must be selected. Local staff, such as the teacher or a counselor should assist the student 
in selecting these goals. The selection of a goal is important, since the NRS will assess 
program performance by comparing students’ outcomes to their stated goals. For 
example, in examining employment measures, a rate or percentage will be computed by 
dividing the number of students who obtained a job with the number of students who 
designated obtaining a job as a goal. The main and secondary goals to be used are: 

S Obtain a job 
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S Retain current job 
S Improve current job 

S Earn a secondary school diploma or GED certificate 
S Enter postsecondary education or training 
S Improve basic literacy skills 
S Improve English language skills 
✓ Obtain citizenship skills 
S Achieve work-based project learner goal 
S Other personal goal 

The following five student status measures are optional. These measures identify special 
target populations identified under WIA. 

❖ Low income status — Record whether the student has low income status by determining 
whether student’s total family income is no more than 70 percent of the Federal income 
standard for a family of his or her family size. This information can be self-reported or 
obtained from documentation. Homeless learners or learners in families receiving cash 
public assistance such as TANF or Food Stamps, are automatically considered low 
income. 

<♦ Displaced homemaker — Designate a learner as a displaced homemaker if he or she has 
been providing unpaid services to family members in the home and has been dependent 
on the income of another family member, but at enrollment is no longer supported by that 
income. The learner must also be unemployed or underemployed and having difficulty 
obtaining a job. Obtain this information through self-report or documentation from 
referring agency. 

❖ Single parent status — Record whether the learner has sole custodial support of one or 
more dependent children from self-report or documentation. 

❖ Dislocated worker status — Designate a learner as dislocated if he or she has been laid 
off of a job or has received notice of impending layoff. Obtain this information from self 
report or documentation. 

❖ Learning disabled adult — Designate a learner as a learning disabled adult if he or she 
has deficits related to neurological impairments in capacity in defined limited areas 
related to learning. Obtain this information from self-report or documentation. 
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Procedures for Collecting Participation Measures 

The NRS participation measures are contact hours and program enrollment type. Teachers or 
other program staff typically collect contact hours and consequently the NRS requirement for 
reporting these hours should have little impact on existing procedures. Local programs have also 
collected program enrollment type, usually at intake. The collection of this measure allows student 
demographics and outcomes to be examined separately for the special program types designated in 
the measure (e.g., workplace and family literacy). 



Contact Hours 

Contact hours are the number of instructional hours the learner receives. All time the learner 
spends in a program-sponsored activity designed to promote student learning in the curriculum 
should be counted. Activities to count include: time in class receiving instruction; time in a learning 
lab, such as a computer lab; time spent with a program-sponsored tutor; and time on assessment 
activities, including initial assessment and follow-up tests, such as post-tests. Time should be 
reported as total hours, rounded to the nearest whole hour. Note that time spent on assessment can be 
counted only if the assessment is designed to inform placement decisions, assess progress or inform 
instruction. Time used to take the GED tests, for example, should not be counted as instructional 
activity. 



Teachers or other instructional staff in the classroom are normally responsible for recording 
attendance hours, although students can also directly record their attendance through a classroom log. 
Attendance should be recorded individually for each student every time the class meets and for all 
adult education classes in which the student is enrolled. 

Teachers should submit contact hours on a regular basis for entry into the program’s data 
management system. Frequent reporting, such as weekly or monthly, is recommended to reduce the 
possibility of error. States are responsible for setting and verifying procedures for reporting contact 
hours. 



Program Enrollment Type 

Indicate the program component in which the student is enrolled. This designation allows for 
separate analyses of students enrolled in individual program areas. Note that ABE, ASE and ESL 
enrollments s implicitly recorded through placement in an educational functioning level. Enrollment 
in other program components must be explicitly recorded. 

❖ Adult basic education (ABE) 

❖ Adult secondary education (ASE) 

❖ English as a second language (ESL) 

❖ Family literacy 

❖ Workplace literacy 
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<♦ Program for the homeless 

❖ Correctional facility 

❖ Community corrections program 

❖ Other institutional program. 

Collecting the NRS Follow-up Measures 

The NRS follow-up measures are outcomes that students may achieve at some time following 
participation in adult education. Both the required core measures and optional secondary measures 
include follow-up measures, which are: 

Core Follow-up Measures 

❖ Entered employment — whedier the student obtained a job. 

❖ Retained employment — whether the student remained in the job. 

❖ Receipt of a secondary school diploma or GED certification. 

❖ Placement in postsecondary education or training. 

Secondary Measures (Optional) 

❖ Reduction in receipt of public assistance — whether the student’s TANF or other public 
assistance was reduced or eliminated due to employment. 

❖ Voting behavior — whether the student registered to vote or voted for the first time. 

❖ Involvement in children’s education — such as involvement in school and helping with 
homework. 

❖ Involvement in children’s literacy activities — such as reading to children. 

❖ Involvement in community activities — such as volunteering to work on community 
projects or joining community groups. 

Note that the secondary, optional measures may also be collected through direct program 
report methodology if the local program obtains the information while the student is still enrolled. 

It is not necessary to collect all of the follow-up measures on all students. Students’ primary 
and secondary goals indicate which follow-up measure applies to them. For example, the entered 
employment measure only applies to students whose goal was to get a job, while measure of receipt 
of a secondary school credential would only apply to learners for whom attainment of this outcome 
was a goal. However, depending on the follow-up methodology, it may be easier to collect all 
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measures on all students and then later match outcomes with goals through the program’s 
management information system. The decision on which procedure to use is left to state discretion. 

The NRS offers two methodologies for collecting these measures: a local program follow-up 
survey and data matching. The local follow-up survey is a survey conducted on all or a random 
sample of learners in each of the state’s adult education programs. Local programs, the state or a 
third party contractor conducts the survey on learners who had a goal of obtained employment in the 
first quarter after the exit quarter for the employment measures. Retained employment is collected in 
the third quarter following exit. The other follow-up measures may be collected any time up to the 
reporting deadline (December 31). States can also use the local survey to collect the secondary 
outcome measures. 

The second methodology is data matching. Under this approach, agencies serving common 
adult education clients, such as education, labor, human service and higher education, pool their data 
and student records are matched on the pooled databases using Social Security numbers. The 
matched data can identify which adult education students achieved the core outcomes — such as 
obtained or kept a job, placement in postsecondary education or obtained a secondary school 
credential. States can also use data matching to collect the secondary measure of reduction in 
welfare benefits. 

Within the NRS, states may use either methodology to collect follow-up measures or a 
combination of the two methods. For example, measures of GED attainment could be collected by 
data matching and the remaining measures could be collected by survey. The following section 
describes the general requirements or models for employing the two methods, including a model 
procedure for conducting the survey and data matching models. Appendix B also contains model 
surveys and guidance on how to conduct a survey. 

Guidance for Conducting the Local Follow-up Survey 

Section 231(e)(2) of WIA requires that the states assess local program performance on the 
core WIA indicators, which are the four NRS core follow-up measures. Consequently, states must 
obtain these measures on students in each of their adult education programs. States electing to 
collect follow-up measures through a local survey (as opposed to data matching) should followed the 
procedures summarized below. Consult Guidelines for Conducting the Follow-up Sur\>ey for more 
detail on survey procedures. 

Universe or Sample Survey 

It is advisable to include all students in a follow-up survey — that is, programs should include 
the universe of learners. For the programs serving large numbers of students with follow-up outcome 
goals, however, the NRS does allow states the option to include a sample of these learners in the 
programs for the survey. 

Sampling a group of students can be much less expensive than a universe survey, but is 
problematic to the degree that the response from the students to the survey is expected to be low. 
Sampling creates a degree of uncertainty or error in the findings from the survey - known as 
sampling error - which becomes quite large if response rate is low. Consequently, the lower the 
response rate, the more difficult it is to make an estimate of the true value of the outcome measure 
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for all students. Since the response rate for adult education students is expected to be low, including 
the universe of learners in the survey rather than a sample is advantageous, since without samplng 
there is no sampling error. However, since most states and adult education programs have minimal 
resources to conduct a survey, the number of students to include needs to be kept a to a minimum, 
making sampling attractive for large programs. 

With these considerations in mind, the NRS guidelines are for programs not to sample 
students, but to survey all students in each outcome goal area that has 300 or fewer students. That 
is, all students should be identified by goal area (entered employment, retain or improve 
employment, enter postsecondary education, obtain secondary credential) and if the total number of 
students for any outcome is 300 or less, all students in that group should be surveyed to determine 
whether they have achieved the appropriate outcome. The minimum acceptable response rate for 
the survey is 50 percent. Programs may sample students for the survey for any outcome group that 
has more than 300 students exiting during the year and must also achieve a 50 percent response rate. 

For example, if a program has 200 students with a goal of obtaining employment, all of these 
students who exit are to be followed to determine whether they obtained employment in the first 
post-exit quarter. If that same program had over 300 students in other follow-up goal areas, the 
program has the option to survey all of these students or to draw a sample of them. In both cases, the 
program must achieve at least a 50 percent response rate. 

Time Period for Conducting the Survey 

The survey may be conducted by the state, local programs or a third-party contractor, as long 
as a program-specific sample is used. The entered employment measure must be collected on 
students by the end of the first quarter after the exit quarter for students who leave the program. A 
job obtained while the student is enrolled can be counted for the entered employment measure, but is 
reported in the first quarter after the student exits. Retained employment must be collected on 
students who had an employment goal and obtained a job by the end of the first quarter after exit 
(including students who obtained a job while enrolled), and students who were employed at 
enrollment and had a goal of improved or retained employment. This measure should be collected in 
the third quarter following the exit quarter on these students. 

Note that retained employment is not reported for students exiting in the third or fourth 
quarter of the program year (i.e., quarters ending March 31 and June 30). Since retained employment 
is measured in the third quarter after exit, there is insufficient time in the reporting period (ending 
December 31) to survey these students. States should report data only for the first two quarters. The 
percentage of retained employment found in these two quarters is assumed to reflect the entire year. 

Exhibit 5 summarizes the times in which data are to be collected, and the student population 
to which each core follow-up outcome measure applies. Students who enroll with a goal of obtaining 
a job must be surveyed in the first quarter after the exit quarter to determine whether they obtained a 
job. These students who obtain a job, and students who were employed at enrollment and had a goal 



* For entered and retained employment, only students who exit are included. For the other follow-up measures, all 
students with follow-up outcome goals should be included in the follow-up for program year 2000. However, 
beginning July 1, 2001, only students who exit during the program year should be included in the follow-up. 
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of improved or retained employment, must then be surveyed in the third quarter after exit to 
determine whether they are still employed. There are no time periods tied to the other follow-up 
measures and they may be collected at any time until the end of the reporting period (December 31). 

Since the entered and retained employment measures are tied to calendar quarters, the 
simplest time to conduct the survey is quarterly. If quarterly collection is conducted, the survey 
should begin no sooner than the last month of the quarter and be completed within three months (one 
quarter). The attainment of postsecondary credential and entering postsecondary education measures 
are not time-bound, so while they can be collected at any time during the reporting period, the easiest 
option is to also collect them by quarter. The program or state should determine the optimal time to 
collect these measures. For example, it may be advisable to collect the entry into postsecondary 
measure in the fall quarter when most students enter community college. If there are scheduled times 
when the GED tests are given, the program could measure that outcome for students with the 
secondary credential goal around that time. It is recommended that equal numbers of students be 
surveyed each quarter. For example, if 300 students are to be surveyed, about 75 students should be 
surveyed each quarter. 

While quarterly data collection is strongly recommended, states may set other times to 
conduct the survey, if they are more convenient or cost-efficient. For example, the program could 
conduct continuous, ongoing surveys or could conduct the survey biannually (such as in December 
and June). However, the time lag to contact students after they exit the program should be as short as 
possible. The longer the time, the lower the response rate (since some students will move) and the 
less valid the data are likely to be. 



Exhibit 5 

Student Population and Collection Time for Core Outcome Measures 



Core Outcome Measure 


Student Population to Include 


Time Period to Collect Measures 


Entered employment 


Learners unemployed at entry with 
employment goal 


First quarter after exit quarter* 
employment while enrolled. 


Retained employment 


Learners unemployed at entry with 
employment goal who obtain a job 
during first quarter after exit; and 
learners employed at entry with a 
goal of retained or improved 
employment 


Third quarter after exit quarter; or 
third quarter after student obtained a 
job, if job obtained while enrolled. 
Not measured for students exiting 
the third or fourth quarter of the 
program year. 


Placement in postsecondary 
education or training 


Learners with a goal of entering 
postsecondary education or other 
training 


Any time to the end of the reporting 
period (December 31). 


Receipt of secondary diploma or 
GED 


Learners with a goal of obtaining a 
secondary diploma or passing GED 
tests 


Any time to the end of the reporting 
period (December 31). 



*Exit quarter is the quarter when the learner completes instruction or has not received instruction for 90 days and has no 
instruction scheduled. A job obtained while the student is enrolled can be counted, but is reported in the first quarter after exit 
quarter. 
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Methodology 

Surveys are conducted by mail, telephone or through an in-person interview. The latter 
option is extremely costly and is not feasible for adult education programs. A mail survey, while 
inexpensive and very easy to administer, normally produces an unacceptably low response rate — 
typically around 10 percent. Consequently it is highly recommended that local programs conduct the 
follow-up survey by telephone. The telephone survey may be supplemented with a mail survey and 
by information collected while the student is enrolled. For example, programs could supply students 
who had an employment goal with a postcard to return when they obtain a job, or the teacher could 
obtain this information while the student is still enrolled. Programs may conduct only a mail survey, 
at the state’s option, if a 50 percent response rate can be achieved. 

It is very important to the success of the survey that students know they may be contacted 
later and asked about their outcomes. In addition, one of the problems in conducting a survey is 
finding the students after they leave the program. It is critical to the survey validity that all students 
in the survey are located, especially if sampling is used. For this reason, programs should inform 
students upon entry into the program about the survey and collect extensive contact information 
about them, such as addresses and phone numbers of relatives or others who may know the students’ 
whereabouts over time. In addition, students should be encouraged to provide new addresses and 
phone numbers when they move and programs should implement procedures to update this 
information periodically while the student remains enrolled. These procedures will greatly assist 
locating students months later when the survey is conducted. 

Appendix B describes model procedures for conducting a telephone survey designed to 
collect the NRS follow-up measures. The model is offered as guidance to states in designing and 
conducting the follow-up survey. The procedures are not required, however, and states are 
encouraged to develop their own procedures for conducting the sur\’ey, as long as they meet NRS 
requirements, conform to accepted scientific practice in the conduct of surveys and result in a sample 
of sufficient size to make meaningful inferences. 

Survey Reporting 

The state must report to the U.S. Department of Education (ED) the overall state percentage 
of students who achieved each of the follow-up outcomes. As required under WIA, ED will report 
these measures to the Congress as indicators of the overall performance of the adult education 
program. The measures are also one factor that ED will use to determine the state’s eligibility for 
incentive awards available under WIA. 

To compute the state overall measures for each outcome, the state will have to aggregate each 
of the measures from every local program to compute a state average. Each local program must 
report the following information to the state to enable computation of the state average. 

• Total number of students in each outcome group served during the year (for employment 

measures, the number who had an employment goal and exited. Beginning in the program 

year that starts July 1, 2001, report only the exited students for all follow-up measures.); 

• If the program sampled, the total number of students sampled; 
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• Number of students who responded to the survey (the realized or actual sample size) and the 
response rate; and 

• The percentage of students achieving each outcome. 

Using the individual program percentages, the state must compute and report the weighted state 
average on Table 5 (see Chapter V). See the NRS document Guidelines for Conducting the Follow- 
up Survey for complete information about conducting the survey and reporting the results. 



Guidance on Data Matching 

Data matching refers to the procedures where two or more state agencies pool and share data 
on a common group of participants. The data consist of individual student records collected by each 
of the agencies that can be linked through a common identifier, typically a Social Security number. 
Matching the pooled data using the common identifier produces a new individual student record or 
an aggregated data report containing data from one or more of the additional agencies. Each agency 
can use the new, pooled data records or reports to understand the impact of their program on 
participants and to obtain data to meet their reporting and accountability requirements. 

Data matching methods are particularly well suited for studying outcomes that occur some 
time after program participation. For example, wage record information systems have been used to 
study the outcomes of vocational education and employment programs. The WIA requires job 
training programs funded under Title I to use a data matching methodology to obtain the required 
employment outcomes. For the NRS, this methodology holds considerable promise for studying the 
core follow-up measures. Among the reasons making it attractive are its cost relative to surveys, 
decreased data collection burdens, and increased data validity. 

The first major advantage of data matching is that it is significantly less costly than the local 
survey methodology. The cost of conducting a survey — drawing a sample, training interviewers, 
making phone calls — are replaced with the much reduced cost of combining, cleaning and analyzing 
the data. Further, this cost can be divided among the participating agencies. 

The second major advantage of data matching is reduced data collection burden. At the local 
program level, staff no longer needs to conduct survey procedures. Local programs would collect 
only the demographic, participation and educational functioning level information. Matching is then 
done at the state level. Finally, matched data are likely to be more valid than those collected through 
surveys, which are self-reported data. Some former students may not be truthful about whether they 
obtained a job, for example. The wage or unemployment record database, however, would reveal 
whether students are working. In addition, response rates for surveys are typically low, so that no 
information is available on a substantial percentage of students. With data matching, considerably 
fewer students are missed, provided each agency has valid Social Security numbers. However, the 
need for Social Security number makes data matching problematic in some states due to 
confidentiality issues. Some states have laws against interagency sharing of Social Security numbers 
and some students are reluctant to give such information to government agencies. 
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Data Matching Models 

Under a data matching system, each participating agency collects a common core of 
demographic and descriptive information on their participants, dates of program participation, a 
common identification number (Social Security number) and the outcome measures specific to its 
program. All measures that are shared among the agencies need to have common definitions if the 
resulting analyses and reports are to be meaningful across agencies. 

There are two data matching models. Under the central data processing or data warehousing 
model, each agency submits its individual client records containing the data to be shared to a central 
source (either a contractor or in-house agency). This central agency combines the information into a 
single data pool and eliminates record duplication using Social Security numbers. This data pool is 
then available to the individual agencies, which can request specific tables and reports. The reports 
are usually in aggregate form at the state, program and site level although individual data reporting 
can be produced. Local program providers can also request reports through their agencies. Exhibit 6 
shows this model of data matching. 



Exhibit 6 

Example of Shared Interagency Data Base— Data Warehouse 




Under a second, decentralized or data harvesting model of data matching, each agency 
maintains its own data records and each separate agency requests matches from the agency with the 
needed data. To match with an outside agency, the requesting agency sends records containing 
Social Security numbers and other data needed for the analysis to another agency, along with the 
format of the data tables needed. The other agency makes the matches and reports the data in the 
requested format. For example, to obtain the GED test results of students, the state could send the 
Social Security numbers of students who had a goal of passing the GED tests, along with 
demographic and program information, to the state agency that does GED testing. The testing 
agency would match the records to produce a report on the number and characteristics of students 
who passed the GED tests. The state could then use this information in its annual NRS reporting. 

For both types of data matching, incorrect or missing Social Security numbers affects the 
availability of data. This problem can be substantial if students refuse to provide their Social 
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Security number or provide incorrect numbers. Legal barriers to collecting Social Security numbers 
also pose a significant barrier to this methodology. Another serious problem affecting data analyses 
with data matching is the time lag from the end of the reporting period until when the data are 
available. It often takes two or more quarters for all of the data to be available in agencies. In states 
using data matching, the time lag ranges from one quarter to a year. For example, if a student leaves 
the program in Februaiy, the entered employment outcome would need to be measured in the next 
quarter (April-June). However, if the time lag is two quarters, that student’s entered employment 
could not be determined until the first quarter of the following calendar year. 

Implementing Data Matching 

Several states currently use data matching procedures to collect outcomes on adult education 
students. This section provides a discussion of some of the major issues that need to be addressed to 
develop data matching to collect the NRS follow-up measures. Further information on developing 
data matching is available in Report on the Pilot Test of the National Reporting System. 

Data matching arrangements are difficult to establish and require considerable time to 
implement. The states using data matching took from two to over five years to implement the 
process. Crucial to implementation of the methodology is an interagency planning process with 
individuals committed to seeing the system development. This process is successful when political 
concerns are kept out of the planning and development. Another essential ingredient is for each 
agency to have an automated, individual participant record system. It is not necessary, however, that 
each agency use the same record system or software, only that the software used by each agency can 
produce information in a common format to allow data matching. 

Beyond these basic planning and infrastructure needs, there are three conceptual problems 
that need to be surmounted to develop shared data arrangements: 

❖ the need for common outcome and measure definitions, 

❖ the need to address concerns about data confidentiality, and 

❖ the need for high quality training and technical assistance. 

The MIS must have common definitions for measures that will be shared. Agencies with 
jurisdiction over different types of programs (for example, Departments of Labor and Education) 
must provide data that are based on common understandings of the measures. Furthermore, agencies 
within a single department (for example, community colleges and local education agencies) must 
also use common definitions. Care must be taken, however, to ensure that the definitions agreed 
upon maintain their fidelity with the mission of the program. If common agreement on common 
definitions cannot be reached, each agency must understand what the other definitions are and must 
be able to accommodate these differences in interpreting the data. For example, if “program 
completion” is a common data element, each agency must use the same definition or must have an 
understanding of what the other definitions are and interpret the data accordingly. 

The issue of confidentiality looms large over data matching procedures. States using data 
matching had to resolve the prohibition about sharing Social Security numbers across agencies. 

States have resolved these difficulties by defining the data matching as research, by allowing only 
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aggregated reporting so that individual students cannot be identified, and/or by obtaining a waiver or 
permission from students. 

Many states have laws against not only sharing Social Security numbers, but against sharing 
educational records. These barriers must be resolved legally before data matching can become a 
widely used methodology for the NRS. 

Finally, a great deal of training and technical assistance at the local level is needed to develop 
a system that produces valid and reliable data. Training needs to be provided on measure definitions, 
data collection and reporting, and data use. Such training also produces buy-in to the whole data 
collection and analysis process and can help “convert” teachers, local staff and other stakeholders 
who might be skeptical about the usefulness of the system. The training can also provide local 
providers with an idea of how the data are used at the state level, as well as how they can use it to 
improve their program. 



Quality Control Guidance 

The data in the NRS are only useful if they are valid — measure what they are supposed to 
measure — and reliable — collected in the same way by different people and at different locations. To 
obtain valid and reliable data, the data collectors must understand the measures and follow the proper 
procedures for collecting the measures at all times with all people. States are responsible for 
promoting data quality and implementing training and quality control procedures for NRS measures. 
This section provides a brief overview of quality control methods that can be implemented prior to 
data collection, during data collection and following data collection. 

Prior to Data Collection: Training 

Within the NRS, the data collectors are local program staff. It is thus critical to the success 
of the NRS that teachers and other local staff involved in collecting and reporting data be provided 
staff development on the NRS. This training should occur prior to the implementation of the NRS 
and should be ongoing, so that initial and follow-up training is available and training is available for 
new staff. Critical topics for training include definitions of measures, completing reporting forms, 
conducting assessments and follow-up methods. 

Understanding and correct use of state assessment procedures are critically important to NRS 
data quality, given the central importance of the educational gain measure. Accurate reporting of this 
measure requires local staff to implement the state assessment methods for intake and progress 
assessment. For example, the progress assessment must be administered at the appropriate time 
determined by the state and staff must follow standardized procedures. Failure to follow the correct 
procedure for administering a standardized test, for example, invalidates the test results. 

While training should cover the general procedures and methods of the NRS, additional 
training on the importance of data and how to use it are likely to increase data quality. When local 
staff can see how to use data for their own purposes, their data collection activities will be more 
meaningful and they are likely to take more care in conducting them. 
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Quality of data will also be enhanced if there are resources available, including state or other 
local staff for local programs to consult when questions or difficulties arise. Through the NRS, 
DAEL has developed a Web-based training site, NRS Online , to assist states in training local staff 

Quality Control During Data Collection 

During the data collection process, there are four mechanisms states and local programs can 
implement that can help ensure data quality. First, data collection procedures need to be explicitly 
organized. Program staff should establish specific, concrete procedures for data collection and data 
reporting. These procedures should state what is to be collected, when it is collected and who is 
responsible for collecting it. The time when the information is to be collected and reported should 
also be determined. Incorporating these procedures formally into staff job responsibilities will also 
enhance the likelihood that staff performs them. 

A second factor critical to collecting quality data is to devote sufficient resources — time, staff 
and money — to data collection. Providing resources signals to staff that data collection is a valued 
and important activity — not something that is done as an afterthought or if there is time. At least one 
staff member in a program should have explicit responsibility for ensuring data are collected and 
reported. 

Reporting data in a timely manner — according to a fixed, regular schedule — is a third factor 
to promoting data quality. Data should be reported to a central agency, such as the state or district, 
frequently and at fixed time periods. At the local level, information should be entered into the 
program’s management information system as frequently as possible. For example attendance 
should be reported weekly or monthly. For reporting to the state, monthly or quarterly reporting is 
highly preferred. If the time lag for reporting data is too long, the data will not be reported 
completely, as staff will have a tendency to put off data reporting until the deadline. The result will 
be a high degree of missing and possibly false data. Another reason for frequent reporting is that 
errors or problems will be identified and can be corrected on an ongoing basis. If data are reported 
only once or twice a year, it would not be possible to identify errors before it was too late to correct 
them. 



Finally, frequent contact with staff that collects data and spot-checking of their data will also 
assist in ensuring quality data. A state or local staff member knowledgeable of reporting and data 
collection should provide regular, ongoing monitoring of data collection through scheduled contact 
with local staff. Samples of data collection forms should be examined periodically. To be most 
effective, monitoring should be proactive and non-punitive, and viewed as a form of technical 
assistance. With this approach, staff is less likely to try to hide problems or cover-up mistakes. 

Quality Control Following Data Collection: Auditing 

States should conduct formal audits of local program data to determine data quality and 
ensure that data procedures have been followed. States can perform data audits as part of regular 
monitoring visits or should schedule them separately. A sample of programs can be selected for 
monitoring annually. Since auditing is done on already collected data, findings from the audit will 
identify problems and technical assistance and training needs that will alleviate the problems in the 
future. 
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During the monitoring visit, a sample of records should be drawn and examined for 
completeness and accuracy. If possible, written records should be compared with information that is 
in the program’s management information system to ensure correspondence between the sources. 
Data collection staff should be interviewed on procedures, particularly on how they deal with missing 
and incomplete information, data entry procedures and reporting times. Another procedure is to 
verify data on currently enrolled students by selecting a sample of students to interview to determine 
whether information in the program’s database is accurate. Monitors should also review the 
program’s assessment and follow-up procedures to ensure they comply with state policy. Exhibit 7 
summarizes the guidance on quality control presented in this section. 

Exhibit 7 

Summary of Quality Control Guidance 



Prior to Data Collection: Training 

❖ Initial training of all staff on NRS definitions and procedures 

❖ Training on state reporting forms, assessment and follow-up policies 

❖ Training on using data for program management and improvement 

During Data Collection 

❖ Organized data collection and reporting procedures with specific staff responsibilities 

❖ Sufficient resources and time provided— a valued activity 

❖ Regular, timely reporting of information 

❖ Proactive outside monitoring and assistance 

Following Data Collection: Auditing 

❖ Periodic auditing of records and procedures 

❖ Interviews with students 

❖ Interviews with data staff 



Quality Control: Implementation Checklist 

To assist states in auditing NRS implementation, Exhibit 8 summarizes the systems and 
procedures required in six areas for collecting quality data. The exhibit is organized as a checklist 
that states can use to review state and local program procedures. State and local programs can also 
use the checklist as a self-assessment tool to identify areas of strength and weakness in 
implementation of NRS requirements and procedures. 

• Student intake and orientation. Since identification of student goals or reasons for attending is 
critical to the collection of NRS follow-up measures, local programs need to have included in 
their intake process procedures for identifying student goals. Intake procedures also should 
include an explanation to the student that the program may survey them to determine whether 
they met their goals. 

• Student assessment. As explained in Chapters II and IV, states must establish statewide, 
uniform assessment procedures. A review of local programs should include whether they are 
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Exhibit 8 

NRS Implementation Checklist for Quality Control 



Student Intake and Orientation 

□ Goal setting procedures for identifying student goals related to core measures have been developed 

□ Procedures for informing students of follow-up procedures have been developed, including use of Social 
Security numbers, if applicable 

Student Assessment and Outcomes 

□ State has established procedures for student assessment that specify: 

□ type of assessment and 

□ criteria for conducting post-test assessments (e.g., time) 

□ Assessments are valid and appropriate for the population being tested and the instructional approach 

□ State has identified NRS secondary (optional) measures to include, if any 

Student Record Keeping 

□ Electronic individual student record system in place 

□ System organized in a relational database to allow for: 

□ Disaggregation to instructional site or classroom level 

□ Identification of individual students by goal for follow-up 

□ Edit checks for test scores and other data integrity 

□ System capable of producing required federal state and local reports 

□ System capable of conducting analyses for program improvement 

□ System capable of being used in interagency data match, if appropriate 

Data Collection Policy and Procedures 

□ Data collection procedures for demographic, assessment and attendance have been established 

□ Clear description of staff roles and responsibilities in data collected have been delineated 

□ Sufficient fiscal resources have been budgeted 

□ Data reporting timeline (quarterly or more frequently) has been established 

□ Follow-up strategy (data matching or survey) has been identified 

□ Quality control procedures (such as onsite data checks) have been established 

Using Data 

□ Plans for how data are to be used have been developed 

□ Formats for reporting have been established 

□ State and local staff use data for program planning and improvement 

Staff Development 

□ Staff have been trained in data collection procedures and using data 

□ Mechanisms for ongoing support have been established 

□ A system for ongoing professional development on data procedures has been established 
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following these procedures appropriately, including using appropriate assessments and post- 
testing at state designated times. 

• Student record keeping. Local programs should have individualized student record systems, 
organized to allow analyses of student, program, site and ideally, class differences. Data 
procedures should include edit checks for data integrity and systems should be capable of 
producing state-required reports. 

• Data collection policy and procedures. Local programs need efficient, well-organized 
procedures for collecting, entering and reporting data. Procedures should include clear 
designation of staff responsibilities, budgeting of appropriate time and resources, and clear 
timelines (monthly or quarterly) for data entry and reporting. There should also be similar 
procedures at the state level. 

• Using data. Effective data collection processes work well when there is a clear understanding of 
how the data will be used. States and local programs need plans for using the data not only for 
reporting, but also for their own program improvement efforts. Appropriate reporting formats 
and procedures should also be in place, 

• Staff development. States and local programs should train staff at all levels in data collection 
procedures and the reason for them. A system of ongoing training needs to be in place due to 
staff turnover. 
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Collection of NRS data produces a rich source of information about adult education students 
and their outcomes. States and local programs can use these data for program accountability, to 
identify effective programs and instruction, and to foster program improvement. Sections 212(c), 
231(e)(2) and 212(a) explicitly identify these purposes in stating the reasons and uses for the program 
accountability system. States must report their performance levels on the core measures to the U.S. 
Department of Education, use the measures to assess the effectiveness of local programs and to 
promote continuous program improvement. 

This chapter provides general guidance on establishing a statewide student reporting system 
that will allow states to meet WIA requirements. The guidance includes a brief summary of the 
software needs and requirements, a description of the information that must be entered into the 
student record system and the types of outputs or reports that states and local programs should be 
able to produce. Concluding this chapter are tables for reporting NRS data to the Federal level. 



General Software Requirements 

To meet NRS requirements, each local program mast use an automated individual student 
record system to enter NRS data. The software of this system must have a relational database 
structure , whereby information on individual students can be related to other variables in the 
database and data can be aggregated and analyzed for specific subgroups. The software must also be 
capable of aggregating data to produce the required Federal reporting tables or the data must be able 
to be imported into other software that will produce the Federal tables. 

The NRS does not require any specific software product or system beyond these 
requirements. States should carefully consider not only NRS reporting requirements, but also their 
reporting needs and the needs and capabilities of local programs in selecting software. Training and 
technical support issues related to software should also be factors in the decision on what software to 
use, as should the overall cost of developing and maintaining the system. Exhibit 9 offers guidance 
on selecting software. 

States should also consider whether software should be distributed individually to each local 
program or centrally maintained and used via the Internet. Centralized approaches make changes and 
enhancements to the software easy to implement and eliminates local reporting, since data are 
directly entered into a central computer. However, central data approaches are often more costly to 
develop and are dependent on using the Internet. Delays in Internet service may frustrate local users. 

Data Structure and Inputs 

The software system should allow local programs to enter and retrieve their own data for 
individual students and to be most useful, the data should be organized by site and class. Exhibit 10 
shows the recommended data structure for NRS reporting and analysis. This structure allows 
programs to examine student outcomes by individual class, by site and for the program overall, and 
thus provides the greatest ability to examine die relationship among instruction and other program 
components and student outcomes. 
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Exhibit 9 

Guidance for Selecting Student Record Software for the NRS 
Issues in Choosing Software 

Programs shopping among the various choices for student record software should consider: 

♦ The overall design of the software; 

♦ The training and support offered by the software’s vendor; 

♦ The methods used to enter data into the software; and 

♦ The various ways that the software allows the program to use data, including reporting, data analysis, and 
program planning functions. 

System Design 

♦ Software issues 

❖ What is the cost of the software? 

❖ Does the software rely on any other software packages in order to function (e g., Microsoft Access)? Do 
local programs have this software? 

❖ What operating system environment is most appropriate for the software (Windows 98 or 95, Windows NT, 
OS/2, etc.)? Is this the system your local programs use? 

♦ Hardware issues 

❖ Do your local programs and sites have computers that are powerful enough and have enough memory to 
run the software (e.g., Pentium-based computers, rather than 486-based machines)? 

❖ Is the software available for PC, Macintosh, or both? Are the data sets interchangeable in a mixed- 
environment setting? 

♦ Usability issues 

❖ Is the software "user-friendly” or intuitive? 

❖ Do potential users appreciate the appearance of the software? 

❖ Can you customize the software to meet your needs? 

❖ Does the software include the specific measures, coding categories, and data elements needed by the 
program? 

❖ Can the software be used for multiple years— are its archives accessible from year to year, or are only the 
current year's records available? 

❖ Can you use the software in a network environment? 

❖ Does the software allow security, such as by limiting access or functionality to specific types of users (e.g., 
password protection, ability to hide sensitive data elements)? 
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Exhibit 9 (Continued) 

Guidance for Selecting Student Record Software for the NRS 

Training and Support 

♦ Does the vendor offer training and/or support? What are the mechanisms of training and support available? 

♦ What is the cost of training or support? 

♦ Does the software have documentation, such as a user’s manual? Is the manual helpful and easily 
understandable? Does it provide useful information? 

♦ Are there planned upgrades for the software? Are software upgrades made available free of charge, and are 
users notified when they become available? 

Data Input 

The ease of entering data into the software can also be an effective way of differentiating among software packages. 

♦ Are data keyed in manually or can it be scanned into the system? 

❖ Is any extra hardware or software necessary for scanning? Can these be leased, or must they be 
purchased? Do all workstations operating the system need these add-ons? 

❖ What scanning mechanism does the software system support (e.g., scantron or bubble forms, text 
scanning). 

♦ What does the software use for a student ID number (e.g., social security numbers or program-defined 
numbers)? Can this number be changed if necessary? 

♦ Does the software support multi-site data entry at the individual program level? Can site-level data be 
aggregated to the program level? 

♦ Can data be imported from other software packages (for example, spreadsheets or other databases)? What 
formats does the software require for imports? 

Using the Data 

Programs should consider how they use data (i.e., for accountablity, program improvement, or program evaluation) 
and whether or not the software addresses these needs. 

♦ Does the software come with built-in reports appropriate for the program’s uses? 

♦ How difficult is it to create reports or modify existing reports as needs arise? Is additional software needed to 
create new reports? 

♦ Does the software allow users to search the database for specific records or conduct queries to locate different 
classes of records? 

♦ What are the analytic capabilities of the software? 

♦ Does the software allow users to conduct analyses at the individual student level? 

♦ Can data be exported to other software packages (for example, spreadsheets or other databases)? What 
formats does the software use for exports? 
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Exhibit 10 

Recommended Database Organization for NRS Reporting and Analysis 




The state and local systems must include, at a minimum, the NRS core measures and the 
coding categories applicable to these measures. Exhibit 1 in Chapter I summarizes the definitions 
and categories and Chapter II provides greater detail on all the measures. In addition to the core 
measures, states that use the secondary measures should include these measures and categories, along 
with any other measures the state needs for its own uses. In addition, states and local programs may 
add coding categories for any core and secondary measures, as long as the NRS categories can still 
be reported. For example, states may use additional functioning levels, categories for ethnicity or 
student goals. To use NRS data to evaluate program performance and promote program 
improvement, the system must also include other measures, such as information about classes, 
instructors and program staff. 

Reporting Capabilities 

Equally important to the system data structure and inputs is the system’s capability to output 
or report information. For Federal reporting, the NRS requires that each state annually submit 
aggregated summary tables of descriptive and performance data on the core measures. To create this 
report, the software of each local program must have the capability to create these reports for the 
program overall and submit an aggregated report to the state, or local programs must be able to 
submit their individual student data to the state for aggregation. States may elect to use either option. 
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While Federal reporting requirements are relatively simple, under WIA states need to use 
NRS data for more extensive purposes. WIA requires states to evaluate each local program’s 
performance on the outcome measures and requires that states address the needs of specific 
subpopulations, such as the low income students or adults in family literacy programs. To obtain this 
information, the software system must have the capability to report by individual program and 
student populations. 

Even more detailed reporting is needed to use NRS data to address program improvement 
needs. Among the most powerful uses of NRS data is the capability to understand the program and 
instructional factors related to successful student outcomes. To study these issues, states and local 
programs need the ability to examine data by site, class and student characteristics, and to relate 
outcomes to such variables as contact hours, teacher characteristics and curriculum. While most 
software systems commonly include such data elements, the reporting of this information in a form 
amenable to program performance evaluation can be problematic unless this capability is built into 
the system initially. 

Exhibit 1 1 presents examples of the types of tables the software should be capable of 
producing. The first three tables in the exhibit show educational advancement by incoming 
educational level, program area and class. State and local administrators could use these tables for 
assessing program performance standards on this measure and to examine which types of students in 
which classes advance at higher rates. The fourth table in the exhibit offers an example of the type of 
report needed to examine individual class data. In this example, student performance is compared to 
full-and part-time instructors and whedier the instructor has had staff development. 



Federal Reporting Tables 

NRS data are to be reported annually to the Division of Adult Education and Literacy 
(DAEL) by each state in aggregate form. DAEL has developed reporting tables for this purpose. 
These tables, which follow this section, have been revised by DAEL reviewed and cleared within the 
Federal government. Instructions on completing each table are included with each table. 

Several optional reporting tables are provided to allow for separate reporting about special 
populations on the core indicators. For example, tables for workplace and family literacy 
participants provide a picture of how their participants performed on core and secondary measures. 
States are encouraged to examine the performance of other target subpopulations separately. 

Two new optional tables are included for the 2001-2002 program year. Table 4 A, added at 
the request of several states, allows states to record die number of students who advanced more than 
one level during the program year, in addition to die number of students who advance one level. 
Table 4B was developed to allow DAEL to examine level completion and advancement separately 
for students who receive both pre- and posttests during the program year. In contrast to the required 
Table 4, which includes all students, states may report data in Table 4B only on students who take 
both a pre- and a post-test. These data allow DAEL to compare the impact of including all students 
with only pre- and post-tested students on educational gains performance levels. Table 4B also 
includes die aggregate number of contact hours by level for the group of pre-and post-tested students 
who advanced. These data will provide an indication of the average amount of instructional time it 
takes for students to advance each level. 
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In addition to data tables, DAEL requires states to submit a narrative report and a financial 
report detailing expenditures. States receiving funds under the EL Civics program must complete a 
separate financial report for the funds. Forms and instructions for both of these financial reports are 
also in the following section. 

States are required to submit the reporting tables six months after the end of the program year 
(December 31). In the future, DAEL will require electronic submission of the tables, will update the 
current reporting software and publish the software technical format when it is available. 
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Exhibit 11 

Sample Tables for Examining Program Improvement 
and Program Effectiveness 



Educational Advancement Information 



III Initial Class Level 


Number Recommended 
for Advancement 


Percentage of 
Students 
Advancing by 
Level 


Avg. Contact 
Hours/Student 
Before 

Advancement 


Beginning Literacy 


21 


12% 


61 


Beginning ABE 


41 


17% 


48 


Low Intermediate ABE 


51 


36% 


39 


High Intermediate ABE 


47 


43% 


40 


LowASE 


23 


38% 


38 


HighASE 


12 


60% 


50 


All Levels 


195 


26% 


46 



Educational Advancement by Program Area 



Program 


Number Enrolled • | 

(all levels) 


Number Recommended ■ 


Percent 


ABE 


225 


58 


26% 


GED 


265 


84 


32% 


ESL 


197 


33 


17% 


Family Literacy 


49 


7 


14% 


Workplace Literacy 


86 


13 


15% 


All Programs 


822 


195 


24% 



Educational Advancement by Class 



• • Class 


Percent Advancing 


fv. T . r n Avg. Contact Hours 

Pre-Test Score Range n * -v-* •••••' ■■■;:■■*> • 

Before Acfyarrcementx: 


Beginning Literacy Class 1 


14% 


162-204 


60 


Beginning ABE Class 1 


17% 


199-214 


51 


Beginning ABE Class 2 


24% 


201-212 


59 


Low Intermediate Class 1 


22% 


209-222 


44 


Low Intermediate Class 2 


31% 


212-219 


39 


High Intermediate Class 1 


26% 


219-233 


42 


All Classes 


22% 


162-233 


49 



BEST COPY AVAILABLE 
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Sample Tables for Examining Improvement and Program Effectiveness 



Observed Using New 
Strategies 
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Participated in 
Professional 
Development on 
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| Full-time 


•> 






























































Low Intermediate Level 


Hrs. Attended 




r"- 
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CO 
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uo 
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CN 

h- 


o 

05 


r^. 

CO 


oo 

h- 


oo 

IN- 


CN 

h- 


CO 


oo 


oo 

o 

CO 


Gain | 




¥ 


¥ 


¥ 


00 


¥ 


r'- 

+ 


¥ 


¥ 


¥ 


1 


CO 


¥ 


o 

T 


¥ 


+ 




¥ 


¥ 


¥ 


t 


¥ 


CN 


¥ 


¥ 


CN 


T 


T 


¥ 


¥ 


in 

CN 

+ 


Post-Test 




O 

CN 

CN 


CNJ 

CNJ 


oo 

CNJ 


oo 

o 

CNJ 


CO 

CNJ 


o 

CN 

CN 


CNJ 


CN 

CN 

CN 


CO 

CN 

CN 


CN 


05 

o 

CN 


O 

CN 

CN 


oo 

CN 

CN 


in 

CN 

CN 


218.4 




r^- 

CN 


CO 

CN 

CN 


^r 

CN 


o 

CN 

CN 


oo 

CN 


CO 

CN 


CO 

CN 

CN 


CN 

CN 


O 

CN 


m 

CN 


CO 

CN 


CN 

CN 


CO 

CN 


217.5 


Pre-Test | 




CN 

CN 


in 

CNJ 


CN 


CN 


CN 


CO 

CN 


CNJ 


r«*- 

CN 


in 

CN 


CN 


CN 

CN 


CN 


co 

CN 


CO 

CN 


213.8 




CO 

CN 


oo 

CN 


CN 


CO 

CN 


in 

CN 


in 

CN 


r — 

CN 


05 

CN 


CN 

CN 


CO 

CN 


CN 


h-. 

CN 


CN 


214.9 


Instructor/Class 


| Name | 


| Barbara Acosta Class #1 


| Angeles, January | 


| Arrendondo, Myra 


| Cassat, Mary 


| Cheswick, Jennifer 


I Dietrich, Greta | 


| Farrar, Allison | 


| Fox, David | 


I Galvan, Bertha 


| Gibson, Corey 1 


| Hadji, Hassan | 


| James, Brad 1 


| Martinez, Juan 1 


| Mulligan, Ivor 


| Simone, Michael 


I Average 


|Stephanie Cronen Class #2 


| Azzam, Rima | 


| Bashir, Lubna | 


1 Burnaska, Kristine 


| Carl, Brad | 


| Escudero, Jaime 


I Hernandez, Maria 


| Patapis, Vicky 


| Portal, Natalie | 


| Rhodes, David | 


| Rodriguez, Hector 


| Sauti, Christina 


| Soden, David 


| Thompson, Terry 


I Average 



CO 

iN 



0 
§- 

* 

CO 

1 



il> 




Measures and Methods for the National Reporting System for Adult Education 




o 

£ 00 




CD 




Exhibit 11 (Continued) 

Sample Tables for Examining Improvement and Program Effectiveness 
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Observed Using New 
Strategies 
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Reading 


O 

z 
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T 
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T 


¥ 
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CO 
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1 


¥ 


¥ 


v 


¥ 
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1 
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CN 
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CO 
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CN 
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CN 


CO 
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Pre-Test 




in 

CN 


CO 

CN 


CO 

CN 


CN 


CN 


CO 

CN 


CN 

CN 


CN 

CN 
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CN 


in 
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CN 
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CN 


| 213.8 | 




in 
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CN 


CO 

CN 


CO 

CN 


in 

CN 


CN 


CD 

CN 


CO 

CN 


co 

CN 


h- 

CN 


CN 


CN 


214.5 


Instructor/Class 


| Name 


| DeWan Lee Class #5 


| Cohen, Crecilla | 


| Cruz, Michelle | 


| DelBorello, David 


| Dowling, Erinn 


| Jiang, Tao | 


| Miller, Patricia | 


| Nesbitt, Daphne 


| Quinones, Sherrie 


| Ramirez, Kevin | 


| Rivera, Jose | 


| Sims, Anthony 


| Taylor, Jessica | 


I Average | 


| Feng Yu Class #6 


| Braswell, James | 


| Carpenter, Daniel 


1 Garcia, Anna | 


| Hall, Pamela | 


| Harper, Sterlina 


| Lopez, Mario 


| Mesmer, Eric 1 


| Olson, Krista 1 


| Rodi, Chad | 


| Sanchez, Anthony | 


| Tanaka, Laurel 1 


| Wagner, Susan | 


| Young, Eboni I 
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Sample Tables for Examining Improvement and Program Effectiveness 



Observed Using New 
Strategies 


o 
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Participated in 
Professional 
Development on 
Reading 


1 No 1 
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Instructor 
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| Full-time 
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Low Intermediate Level 


Hrs. Attended | 




CO 
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CO 
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05 


co 


CO 
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CO 


CO 

r~- 


1 84.8 | 




CO 

I s - 


CM 

CO 


CO 

CO 


CM 


in 


CO 
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CO 


CO 


co 


CJ) 

cri 

h- 


Gain 




1 


CM 

+ 


CO 

+ 


CM 

+ 


¥ 


in 

+ 


¥ 


in 

+ 


00 

CO 

+ 




CM 

+ 


7 


7 


1 


i 


7 


+ 


7 


CO 

+ 


Tf 

+ 


Post-Test | 




cm 


213 | 


in 

CM 


| LIZ 


223 | 


| IZZ 


219 | 


1 218 | 


! 217.1 1 




| 8fZ 


1 219 | 


CM 


CM 


1 216 1 


1 512 


1 216 [ 


1 212 | 


h- 

CM 


215.3 


Pre-Test 




CM 


CM 


1 212 | 


in 

CM 


in 

CM 


CO 

CM 


1 213 | 


1 213 [ 


I 213.3 ! 




CO 

CM 


1 218 | 


213 ! 


CM 


216 | 


1 HZ 


CM 

CM 


CM 


CM 


213.9 


Instructor/Class 


| Name | 


| Jennifer Lewis Class #7 


| Baldi, Stephane 


| Dwyer, Kevin 


| Honegger, Steven 


| Johnson, Tony | 


| Pisacane, Kerry | 


| Rudick, Sherrie | 


| Weidberg, Suzanne 


| Yoon, Kwang [| 


I Average | 


| Arlinda Morris Class #8 1 


| Busch, Melissa 


| Etheridge, Gretchen 1 


| Huang, Yun (Ellen) 


| Jones, Tarsha | 


I Millstone, Ken | 


| Paley, Belen ] 


| Rodriguez, Carlos | 


| Spears, Eric 


| Woodford, Alix 


| Average 
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Reporting Tables 



According to the Paperwork Reduction Act of 1995, no persons are required to respond to a 
collection information unless such collection displays a valid OMB control number. The valid OMB 
control number for this information collection is 1830-0027. The time required to complete this 
information collection is estimated to average 1 10 hours per response, including the time to review 
instructions, search existing data resources, gather the data needed, and complete and review the 
information collection. If you have any comments concerning the accuracy of the time estimate 
or suggestions for improving this form, please write to: Division f Adult Education and Literacy, 
Office of Vocational and Adult Education, U.S. Department of Education, 400 Maryland Avenue, 
S.W., Washington, DC 20202-4651. If you have comments or concerns regarding the status of 
your individual submission of this form, write directly to: Division of Adult Education and 
Literacy, Office of Vocational and Adult Education, U.S. Department of Education, 400 Maryland 
Avenue, S.W., Washington, DC 20202-4651. 
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Table 5 

Core Follow-up Outcome Achievement 



Enter the number of participants for each of the categories listed and calculate the percentage 
achieving each outcome. 



Core Follow-up 
Outcome 
Measures 

(A) 


Number of 
Participants 
with Main or 
Secondary Goal 

(B) 


Number of 
Participants 
Included in 
Survey 
(Sampled and 
Universe) 
(C) 


Number of 
Participants 
Responding 
to Survey or 
Used for Data 
Matching 
(D) 


Response 
Rate or 
Percent 
Available 
for Match 

(E) 


Number of 
Participants 
Achieving 
Outcome 

(F) 


Weighted 

Average 

Percent 

Achieving 

Outcome 

(G) 


Entered 

Employment* 














Retained 

Employment** 














Obtained a GED or 
Secondary School 
Diploma*** 














Entered 
Postsecondary 
Education or 
Training**** 















* Report in Column 6 the number of participants who were unemployed at entry and who had a main or secondary 
goal of obtaining employment and exited during the program year. 

** Report in Column B: (1) the number of participants who were unemployed at entry and who had a main or 
secondary goal of employment who exited in the first and second quarter and entered employment by the end of the 
first quarter after program exit, and (2) the number of participants employed at entry who had a main or secondary 
goal of improved or retained employment who exited in the first and second quarter. Exclude from this total all 
participants who exited in the third and fourth quarters of the program year (see Implementation Guidelines for 
explanation). 

*** Report in Column B the number of participants with a main or secondary goal of passing the GED tests or 
obtaining a secondary school diploma or its recognized equivalent. Effective the program year beginning July 
1, 2001 report in Column B only students with this goal who exited during the program year. 

**** Report in Column B the number of participants with a main or secondary goal of placement in postsecondary 
education or training. Effective the program year beginning July 1,2001 report in Column B only students with 
this goal who exited during the program year. 

InstructionsforcompletingColumnsC-Ediffer according to (1) whether all local programs in the state used a 
survey to obtain the measure and one or more programs sampled students, (2) whether all local programs in the 
state used a survey to obtain the measure and no programs sampled students or (3) whether data matching was 
used to obtain the measure. See instructions below for each alternative and examples under Additional Instructions 
that illustrate completion of the table. 
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1 . All Local Programs in the State Used a Survey for the Measure and One or More Programs Sampled 
Students 

Report in Column C the total number of participants from Column 6 who were surveyed. This number should be the 
sum of the total samples drawn for all programs that sampled and the sum of all students with the goal for programs 
that did not sample. (For guidance on sampling see Guidelines for Conducting the Follow-up Survey and attached 
summary of sampling guidelines). 

Report in Column D the number of participants from Column C who responded to the survey. 

Compute the percentage in Column E for the measure using the following formula: E = — — 'iHHAL 

ColumnC 

2. All Local Programs In The State Used a Survey to Obtain The Measure and No Programs Sampled 
Students 

Column C = Column B for any measure where all programs in the state included all participants with the goal in a 
survey to obtain the measure (i.e., no program sampled). 

Report in Column D the number of participants from Column C who responded to the survey. 



Compute the percentage in Column E for the measure using the following formula: 



ColumnD 

ColumnB 



3. The State Used Data Matching To Obtain The Measure 

Do not complete Column C - leave it blank. 

Report in Column D the number of participants whose records were used in data matching for this measure. 



Compute the percentage in Column E for the measure using the following formula: 



ColumnD 

ColumnB 



All states complete Columns F-G as follows. 



Report in Column F the number of participants from Column D who achieved each outcome. For retained 
employment, enter the number of participants from Column D who were employed in the third quarter after program 
exit. However, exclude from this total in Column D all participants who exited in the third and fourth quarters of the 
program year (see Implementation Guidelines for explanation). 

Each row total in Column G is the weighted state average percent calculated by multiplying the proportion achieving 
the outcome in each program by the total number of participants in each program with the outcome goal during the 
program year. This product is summed across all programs and divided by the totals in Column 6. Note that for the 
rate reported in Column G to be considered valid, the rate reported in Column E must be at least 50 percent. See 
Additional Instructions below for further explanation. 

OMB Number 1830-0027, Expires 1/31/03. 
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Additional Instructions for Completing Table 5 
Computing the State Weighted Average (Column G) and Examples 

States are to report the weighted average for the state for each measure in Column G of Table 5. The use 
of a weighted average is necessary to ensure that large programs contribute to the overall state average in 
proportion to their size (i.e., programs serving more students count more in the average). The weighted average also 
allows adjustment for nonresponse of participants who do not complete the survey or were not included in data 
matching. According to NRS policy, the percentage of participants achieving the outcome among respondents (or 
records matched in a data match) is assumed to be the same as the percentage found among nonrespondents (or 
not included in a data match), if there is at least a 50 percent response rate, In other words, NRS policy allows the 
assumption that if a survey (or data match) finds that 40 percent of participants who were surveyed (or included in a 
data match) entered employment, then 40 percent of participants who did not respond to the survey (or were not in 
the data match) also entered employment. 

The response rate or percentage of records available to match (reported in Column E) is critical to the 
validity of this assumption. For this reason, NRS policy sets a minimum of 50 percent response (or match availabilty) 
for all measures. The Department of Education may question the validity of the reported percentage achieving 
outcome for any measure where response falls below 50 percent. For further information see Appendix A of 
Guidelines for Conducting the Follow-up Survey. 

Computing the Weighted Average 

Computation of the weighted average is simple: for each program, take the proportion of students found to 
have achieved each outcome and multiply it by the total number of students to which the outcome applies (i.e., the 
number of students with main or secondary goal related to the outcome during the program year). This product is the 
estimated number of students in the program who had the goal who achieved the outcome. Sum this number across 
all programs and divide it by the total number of students in the state with the main or secondary goal related to that 
outcome (number in Column B of Table 5 for the outcome measure). The mathematical formula for the weighted 
average is: 

X p>Ns 

P stale = 

Ye Ns 

I- 1 

The following three examples illustrate the computation of the weighted state average for situations where (1) 
local programs use a survey and one or more local programs sample students, (2) where local programs use a 
survey and no programs sample students and (3) where a state uses data matching. Following these examples is a 
sample Table 5 completed with each example in a separate row, to provide further guidance on how to complete the 
table. 

State Example #1 : All Local Programs Used a Survey for a Measure and one or More Programs Sampled 

State A has three local programs and a total of 8,200 students to which the outcome applies (i.e., had the outcome 
as a main or secondary goal). Programs 1 and 2 sampled students and Program 3 did not. 

Program 1 has 4,000 students, samples 500, and reaches 250. 125 of these (50%) achieved the outcome. 

Program 2 has 4,000 students, samples 500, and reaches 250. 125 of these (50%) achieved the outcome. 

Program 3 has 200 students to survey, 100 responded, 30 (30%) achieved the outcome. 
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To compute the state average, the state has to take the percentage found to have achieved the outcome in each 
program (p), multiple it by the total number of students in the program with the goal (n), sum this product across all 
programs and divide it by the total number statewide (N). So for this example: 

Program 1 and Program 2: p=. 5, n=4,000 

Program 3: p=. 3, n=200. Total state N=8,200 

The state average is: (,5)(4,000) + (.5)(4,000) + (.3)(200) 



= .495 or 49.5% 



4000 + 4000+200 



State Example # 2: All Local Programs Used a Survey to Obtain the Measure and No Programs Sampled 

State B has three local programs and a total of 800 students to which the outcome applies (i.e., had the outcome as 
a main or secondary goal during the program year). None of the three programs sampled. 

Program 1 has 300 students to survey, 200 respond, and 80 (40%) achieved outcome. 

Program 2 has 300 students to survey, 100 respond, 30 (30%) achieved the outcome. 

Program 3 has 200 students to survey, 1 20 respond, 40 (33%) achieved the outcome. 

To compute the state average, the state has to take the percentage found to have achieved the outcome in each 
program (p), multiple it by the total number of students in the program with the goal (n), sum this product across all 
programs and divide it by the total number statewide (N). So for this example: 

Programs 1: p=. 4, n=300; Program 2: p=. 3, n=300; Program 3: p=. 33, n=200. Total state N=800 

The state average is: (,4)(300) + ( 3)(300) + (,33)(200) 



~= .345 or 34.5% 



300 + 300 + 200 



State Example# 3: State Uses Data Matching to Obtain the Measure 

State C conducts a statewide data match to obtain the outcome measure. The state has five local programs and a 
total of 1 1,000 students to which the outcome applies (i.e., had the outcome as a main or secondary goal during the 
program year). 

Program 1 has 4,000 students, 3,800 available for matching of which 2,000 achieved the outcome, i.e., are matched 
(52.6%). 

Program 2 has 4,000 students, 3,000 available for matching of which 1,500 achieved the outcome, i.e., are matched 
(50%). 
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Program 3 has 1,000 students, 900 available for matching of which 400 achieved the outcome, i.e., are matched 
(44.4%). 

Program 4 has 1 ,500 students, 1,300 available for matching of which 800 achieved the outcome, i.e., are matched 
(61.5%). 

Program 5 has 500 students, 400 available for matching of which 200 achieved the outcome, i.e., are matched 
(50%). 

To compute the state average, the state has to take the percentage found to have achieved the outcome in each 
program (p), multiple it by the total number of students in the program with the goal (n), sum this product across all 
programs and divide it by the total number statewide (N). So for this example: 

Programs 1 : p=. 526, n=4,000; Program 2: p=. 5, n=4,000; Program 3: p=. 444, n=1,000; 

Program 4: p=. 615, n=1,500; Program 5: p=. 5, n=500. Total state N=1 1,000 

The state average is: (,526)(4000) + (,5)(4000) + (,444)(1000) + (,615)(1500) + (,5)(500) 

= .52 or 52% 

4000 + 4000 + 1000 + 1500 + 500 

Sample Completed Table 5 for Three Examples 

Table 5 is completed for the three examples above to illustrate how a state would complete the table for a measure 
under the different scenarios. 



Core Follow-up 
Outcome 
Measures 

(A) 


Number of 
Participants 
with Main or 
Secondary Goal 

(B) 


Number of 
Participants 
Included in 
Survey 
(Sampled and 
Universe) 
(C) 


Number of 
Participants 
Responding 
to Survey or 
Used for Data 
Matching 
(D) 


Response 
Rate or 
Percent 
available 
for Match 

(E) 


Number of 
Participants 
Achieving 
Outcome 

(F) 


Weighted 

Average 

Percent 

Achieving 

Outcome 

(G) 


State A -All 

Programs 
Surveyed, Some 
Sampled 


8,200 


1,200 


600 


50% 


280 


49.5% 


State B - All 

Programs 
Surveyed, None 
Sampled 


800 


800 


420 


52.5% 


150 


34.5% 


State C — State 
Data Matching 


11,000 




9,400 


85.4% 


4,900 


52% 
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Addendum to Table 5 

Summary of NRS Survey and Sampling Guidelines 

1 . Identifying the Survey Population 

• In each program in the state, identify (1) all unemployed students with a goal of obtaining a job who exited, (2) 
employed students with a goal of keeping or improving their current job who exited, (3) students with a goal of 
obtaining a secondary diploma or passing the GED and (4) students with a goal of entering postsecondary 
education or training. Beginning the program year that starts July 1, 2001, include only students who exited 
during the program year. 

• Obtain the exit date or quarter for students with employment goals to identify follow-up date. 

• Exclude from the retained employment measure all students with an employment goal who exited in the third or 
fourth quarters. 

• If the state is requiring collection of the NRS optional measures, the students with goals related to these 
outcomes should also be identified. 

2. Selecting Students for the Survey 

• Survey all students in any outcome group that has 300 or fewer students in the program year. 

• A simple random sample of students may be surveyed for any outcome group that has more than 300 students. 

• Programs should draw a minimum sample size of 300 for each group that has 301 - 5,000 students. 

• Programs should draw a minimum sample size of 1 ,000 for each group that has over 5,000 students. 

• A minimum response rate of 50 percent is required. If less than 50 percent response rate is achieved, the 
program must survey for any group that was sampled until it has responses from at least 150 students (or 500 
students for the largest programs). 

3. Collecting the Data 

• Quarterly data collection is recommended for all measures, since entered employment must be recorded for the 
first quarter after program exit and retained employment in the third quarter after exit. 

• Retained employment is to be collected only for students exiting in the first two quarters of the program year. 

4. Reporting the Findings 

• Each program must report for each outcome group: the total number of students with the goal; the total number 
of students responding to the survey; the number and percentage of students achieving each outcome; and the 
total number of students included in the survey or if sampling was employed, the total number sampled. 

• The state is to compute a weighted state average of students achieving each outcome by aggregating from the 
individual programs: the total number of students in each outcome group, response rates and the number and 
percentage achieving each outcome. The state must also report the state confidence interval if any local 
programs use sampling. 
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Table 6 

Participant Status and Program Enrollment 
Enter the number of participants for each of the categories listed. 



Participant Status on Entry into the Program 

(A) 


Number 

(B) 


Disabled 




Employed 




Unemployed 




Not in the Labor Force 




On Public Assistance 




Living in Rural Areas* 




Program Type 


In Family Literacy Programs** 




In Workplace Literacy Programs** 




In Programs for the Homeless** 




In Programs for Work-based Project Learners** 




Institutional Programs 


In Correctional Facilities 




In Community Correctional Programs 




In Other Institutional Settings 




Secondary Status Measures (Optional) 


Low Income 




Displaced Homem aker 




Single Parent 




Dislocated Worker 




Learning Disabled Adults 





‘Rural areas are places of less than 2,500 inhabitants and outside urbanized areas. 
“Participants counted here must be in program specifically designed for that purpose. 
OMB Number 1830-0027, Expires 1/31/03. 
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Table 7 

Adult Education Personnel by Function and Job Status 

Enter an unduplicated count of personnel by function and job status. 



Function 

(A) 


Adult Education Personnel 


Unpaid Volunteers 
(D) 


Total Number of 
Part-time Personnel 

(B) 


Total Number of 
Full-time Personnel 

(C) 


State-level Administrative/ 
Supervisory/Ancillary Services 








Local-level Administrative/ 
Supervisory/Ancillary Services 








Local Teachers 








Local Counselors 








Local Paraprofessionals 









In Column B, count one time only each part-time employee of the program administered under the Adult Education State Plan 
who is being paid out of Federal, State, and/or local education funds. 



In Column C, count one time only each full-time employee of the program administered under the Adult Education State Plan 
who is being paid out of Federal, State, and/or local education funds. 

In Column D, report the number of volunteers (personnel who are not paid ) who served in the program administered under the 
Adult Education State Plan. 

OMB Number 1830-0027, Expires 1/31/03. 
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Table 8 

Outcomes for Adults in Family Literacy Programs (Optional) 
Enter the number of participants in family literacy programs for each of the categories listed. 



Core Follow-up 
Outcome Measures 

(A) 


Number of 
Participants 
with Main or 
Secondary 
Goal 

(B) 


Number of 
Participants 
Included in 
Survey 
(Sampled and 
Universe) 
(C) 


Number of 
Participants 
Responding to 
Surveyor 
Used for Data 
Matching 
(D) 


Response 
Rate or 
Percent 
available 
for Match 

(E) 


Number of 
Participants 
Achieving 
Outcome 

(F) 


Weighted 

Average 

Percent 

Achieving 

Outcome 

(G) 


Completed an 
Educational Functioning 
Level * 














Entered Employment 














Retained Employment 














Obtained a GED or 
Secondary School 
Diploma 














Entered Postsecondary 
Education or Training 














Increased Involvement in 
children’s education 

Help more frequently 
with school 

Increased contact 
with children’s 
teachers 
More involved in 
children’s school 
activities 
















































>f !• III!! 


Increased involvement in 
children's literacy 
activities 

Reading to children 

Visiting library 

Purchasing books or 
magazines 
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* Report in Column Sfor this row all family literacy program participants who received 12 or more hours of service. 
Column F should include all participants reported in Column 6 who advanced one or more levels. 

Compute Column Gfor this row using the following formula: G = 

ColumnB 

Follow instructions for completing Table 5 to complete the remainder of this table. However, include only family 
literacy program participants in T able 8. 

Achievement of one or more of the increased involvement in children's education or children's literacy activities 
measures should be counted in this row only once per participant. However, the specific outcome should be 
recorded in the subcategory and more than one outcome may be reported, so that the total for the three 
subcategories may be greater than the total reported for the overall category. For example, a participant who helped 
more frequently with school work and increased contact with child's teachers would be recorded in both categories 
but would be counted only once in the overall category of “increased involvement in children's education.” 

OMB Number 1830-0027, Expires 1/31/03. 
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Table 9 

Outcomes for Adults in Workplace Literacy Programs (Optional) 
Enter the number of participants in workplace literacy programs for each of the categories listed. 



Core Follow-up 


Number of 


Number of 


Number of 


Response 


Number of 


Weighted 


Outcome 


Participants 


Participants 


Participants 


Rate or 


Participants 


Average 


Measures 


with Main or 


Included in 


Responding to 


Percent 


Achieving 


Percent 




Secondary 


Survey 


Survey or Used 


available 


Outcome 


Achieving 




Goal 


(Sampled and 


for Data 


for Match 




Outcome 






Universe) 


Matching 








(A) 


(B) 


(C) 


(D) 


(E) 


(F) 


(G) 


Completed an 




— 


— i— 








Educational 
Functioning Level* 






■1 

MHMNI 






Entered 

Employment 














Retained 

Employment 














Obtained a GED or 
Secondary School 
Diploma 














Entered 
Postsecondary 
Education or 
Training 

i 















* Report in Column 6 for this row all workplace literacy program participants who received 12 or more hours of 
service. Column F should include all participants reported in Column B who advanced one or more levels. 

ColuttiviF 

Compute Column 6 for this row using the following formula: G = — 

ColumnB 

Follow instructions for completing Table 5 to complete the remainder of this table. However, include only workplace 
literacy program participants in T able 9. 

OMB Number 1830-0027, Expires 1/31/03. 
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Table 10 

Outcomes for Adults in Correctional Education Programs (Optional) 
Enter the number of participants in correctional education programs for each of the categories listed. 



Core Follow-up 
Outcome 
Measures 

(A) 


Number of 
Participants 
with Main or 
Secondary 
Goal 

(B) 


Number of 
Participants 
Included in 
Survey 
(Sampled and 
Universe) 
(C) 


Number of 
Participants 
Responding to 
Survey or Used 
for Data 
Matching 
(D) 


Response 
Rate or 
Percent 
available 
for Match 

(E) 


Number of 
Participants 
Achieving 
Outcome 

(F) 


Weighted 

Average 

Percent 

Achieving 

Outcome 

(G) 


Completed an 
Educational 
Functioning Level* 




a 

■ 


mm 


'■;,i 

gillllM 






Entered 

Employment 














Retained 

Employment 














Obtained a GED or 
Secondary School 
Diploma 














Entered 
Postsecondary 
Education or 
Training 















* Report in Column S for this row all correctional educational program participants who received 1 2 or more hours of 
service. ColumnF should include all participants reported in Column B who advanced one or more levels. 

ColumnF 

Compute Column Gfor this row using the following formula: G = — 

ColumnB 

Follow instructions for completing Table 5 to complete the remainder of this table. However, include only correctional 
educational program participants in Table 10. 

OMB Number 1830-0027, Expires 1/31/03. 
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Table 11 

Secondary Outcome Measures (Optional) 
Enter the number of participants for each of the categories listed. 



Secondary Outcome Measures 

(A) 


Number of 
Participants 
with Main or 
Secondary Goal 
or Status 
(B) 


Number of 
Participants 
Obtaining 
Outcome 

(C) 


Percentage 

Achieving 

Outcome 

(D) 


Achieved work-based project learning goal 








Left public assistance 








Achieved citizenship skills 








Increased involvement in children's education* 








Increased involvement in children’s literacy 
activities* 








Voted or registered to vote 








Increased involvement in community activities 









Each row total in Column D is calculated using the following formula: p=^ o/u '" nC 

ColumnB 



* Enter the total number of participants who achieved this goal regardless of whether the participant was in a family literacy 
program. Use Table 8 to enter achievements of family literacy participants. The number reported here may be higher than 
reported in Table 8 since it includes all participants who achieved this goal. 

OMB Number 1830-0027, Expires 1/31/03. 
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(N) 














White 


Female 

(M) 














Male 

(L) 














Native Hawaiian 
or Other Pacific 
Islander 


Female 

(K) 














Male 

(J) 














Hispanic or 
Latino 


0) 

CD 

E 3 

CD 

LL. 














Male 

(H) 














Black or African 
American 


Female 

(G) 












1 


Male 

(F) 














C 

CD 


Female 

(E) 














V) 
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(C) 














CD OZ 

o<5 

1° 

< 


Male 
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1 
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J 
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Age Group 

(A) 


16-18 
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45-59 


60 and Older 
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Table 13 (Optional) 

Core Follow-up Outcome Achievement for 
Prior Reporting Year and for Unintended Outcomes 

For Column B, enter the number of participants for each of the outcome categories for outcomes not 
reported in the prior reporting period. For Column C, enter the number of participants achieving each 
outcome who did not have the outcome as a goal. 



Core Follow-up Outcome Measures 

(A) 


Number of Participants 
with Main or Secondary 
Goal Who Achieved 
Outcome but Were Not 
Reported in the Prior 
Reporting Period 
<B) 


Number of Participants 
Achieving Outcome in 
Current Year Who 
Did Not Have the 
Outcome as a Goal 
(C) 


Entered Employment 






Retained Employment 






Obtained a GED or secondary school diploma 






Placed in postsecondary education or training 







For Column B, report the number of participants who had the core outcome as a primary or secondary goal and who 
achieved that outcome according to the core outcome definitions (see T able 5), but were not reported in the prior 
program year. 

For Column C, report the number of participants who achieved the outcome in the current reporting year but did not 
have the outcome as a main or secondary goal. 
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PART II: NARRATIVE 

Provides descriptive information for the narrative items. 

Maximum number of pages is 10. 

Use the following outline in preparing the narrative part of the Annual Performance Report: 

1. Describe successful activities, programs, and projects supported with State Leadership funds 
and describe the extent to which these activities, programs, and projects were successful in 
implementing the goals of the State Plan. 

2. Describe any significant findings from the Eligible Agency's evaluation of the effectiveness 
of the adult education and literacy activities based on the core indicators of performance. 

3. Describe how the Eligible Agency has supported the integration of activities sponsored 
under Title II with other adult education, career development, and employment and training 
activities. Include a description of how the eligible agency is being represented on the Local 
Workforce Investment Boards, adult education’s involvement on the State Workforce 
Investment Board, the provision of core and other services through the One-Stop system and 
an estimate of the Title II funds being used to support activities and services through the 
One-Stop delivery system. 

4. Describe successful activities and services supported with EL Civics funds, including the 
number of programs receiving EL Civics grants and an estimate of the number of adult 
learners served. (Only states receiving EL-Civics funds should respond to question #4). 
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PART III: FINANCIAL STATUS REPORT 

OMB Number: 1830-0027 Expires: 

U.S, Department of Education 
Office of Vocational and Adult Education 
Adult Education and Family Literacy Act of 1998 
Basic Grants to States — CFDA 084.002 



A separate set of Financial Status Report (FSR) forms are to be used for each Federal Funding Period 

as reported in Block 8 of the FSR for Adult Education. 

Instructions for Completing the FSR. 

Block 

1. This block is preprinted. 

2. PR/ Award numbers as indicated in Block 5 of the Grant Award Notifications for the Basic 
Grants to States. 

3. Grant recipient submitting report. 

4. Enter DUNS/SSN Identifying number in Block 8 of the Grant Award Notification. 

5. For optional use for diose agencies needing cross reference identification. 

6. Check Yes if this is the Final report for a grant award and there are no amounts reported in 
column h (unliquidated obligations). The report is final when there are no additional outlays 
or obligations against the grant award and all existing obligations have been liquidated. 

7. Identify the accounting basis used by the Grantee. If the modified accrual basis is used, it 
should be so indicated by adding the word DmodifiedD in this block. 

8. Enter Federal Funding Period based on information obtained in Block 6 of the Grant Award 
Notification. 

9. Enter the beginning and ending dates of the period in which you are reporting the financial 
activity of the grant. A first year report will cover the first 15 months of the grant period, e.g., 
July 1, 1999 through September 30, 2000. The final report will cover the entire 27 months 
which grantees have to obligate their funds e.g,, July 1, 1999 through September 30, 2001. 

10. The Columns (a) through (f) contain preprinted headings for reporting expenditures. The 
following are explanations of what expenditures should be reported in each column. 
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Column (a). State Administration. Report State administrative expenditures authorized 
section 222 (a)(3) of the Adult Education and Family Literacy Act (AEFLA). 

Column (b). State Leadership. Report expenditures authorized in section 222 (a)(2) 
and described in section 223 of AEFLA. 

Columns (c) and (d), Programs of Instruction. Report all expenditures made by local 
eligible providers in conducting basic education and English literacy (column c), and 
secondary education programs of instruction (column d), including expenditures for 
institutionalized persons. 

Column (f). Institutionalized. Report expenditures for programs for institutionalized 
persons. These expenditures will also appear in columns (c) and (d). 

10a. In the first year report of the grant award, this column must be zero. In the final report, the 
amount reported should be the same as the amount reported on line lOe of the first year 
report made for the same grant award. If there has been an adjustment of the amount shown 
previously, attach explanation. For reports made on a cash basis, outlays are the sum of 
actual cash disbursements for goods and services, the amount of indirect expense charged, 
the value of in-kind contributions applied, and the amount of cash advances and payments 
made to contractors and subgrantees. For reports prepared on an accrued expenditure basis, 
outlays are the sum of actual cash disburesements, the amount of indirect expense incurred, 
the value of in-kind contributions applied, and the net increase (or decrease) in the amounts 
owed by the recipient for goods and other property received and for services performed by 
employees, contractors, subgrantees, and other payees, and otherpayees, and other amounts 
becoming owed under programs for which no currrent services or performances are required, 
such as annuities, insurance claims, and other benefit payments. 

10b. Total outlays, including any state and local outlays, for the report period indicated in Item 9. 

10c. Program credits must be included on this line and are to be used to reduce total outlays. 

lOd. Line b minus line c. 

lOe. Line a plus line d. 

lOf. Enter amount of non -federal outlays reported in line b. 

lOg. Line c minus line f. 

lOh. All unliquidated obligations as of the end of the reporting period. 

lOi-j. Unliquidated obligations are- 

Cash basis - obligations incurred but not paid 

Chapter V. NRS Reporting 94 
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Accrued expenditure basis - obligations incurred but for which an outlay has not been recorded. 

Do not include any amounts that have been included on Lines a through g. Include unliquidated 

obligations to subgrantees and contractors. 

If the report is final, it should not contain anyh unliquidated obligations. 

10k. Line g plus line j. 

101. The amount of Federal funds awarded, per the cumulative amount in Block 7 of the Grant 
Award Notification. 

10m. Line 1 minus line k. 

1 la. Self-explanatory 

lib. Enter the indirect cost rate in effect during the reporting period. If more than one rate was 
applied during the reporting period, include a separate schedule showing the bases against 
which the indirect cost rates were applied. 

1 lc. Enter amount of the base to which the rate was applied. 

1 Id. Enter total amount of indirect cost charged during the reporting period. 

12. Include any remarks necessary to explain any specifics in the report. Attach additional 
information if needed. 

13. The report must be certified by the Executive Officer, or designee, of the Grant recipient, as 
appropriate. 

Reports are due on December 31 each year. Reports should be submitted to: 

Division of Adult Education and Literacy 
Office of Vocational and Adult Education 
Department of Education 
400 Maryland Avenue, SW 
Washington, DC 20202-7240 




117 



Measures and Methods for the National Reporting System for A dult Education 



Put Financial TABLE FORM here (H:\prj\NRS\Task 9\March Guideline Revisions\ 
“Financial Report.doc”) 




II 3 
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Financial Reporting Requirements for EL-Civics Funding 

States expending EL-Civics funds under the conditions outlined in Program Memorandum 2000-19, 
issued by Ronald S. Pugsley on May 16, 2000 shall report those expenditures as follows: 

In addition to submitting an annual Financial Status Report (FSR) reporting all federal and non- 
federal expenditures, including those for EL-Civics, a separate FSR for EL-Civics expenditures is 
also required. This EL-Civics FSR, which represents a sub-total of the overall report, will provide the 
necessary information to determine that EL-Civics expenditures were in compliance with existing 
statutory requirements. A specially identified EL-Civics FSR is included for your use. 
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Put EL CIVICS TABLE FORM here (H:\prj\NRS\Task 9\March Guideline Revisions\ 
“EL civics financial.doc”) 
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NRS Advisory Board 



Judy Crocker 
Cleveland Public Schools 
Cleveland, Ohio 

Garland Hankins 

Arkansas Department of Education 
Little Rock, Arkansas 

G. David Massey 

North Dakota Department of Education 
Bismarck, North Dakota 

Judy McCoy 
Overfelt Adult School 
San Jose, California 

Daniel Miller 

Illinois State Board of Education 
Springfield, Illinois 

Roberta Pawloski 

Comiecticut Department of Education 
Hartford, Connecticut 

Kathy Polis 

West Virginia Department of Education 
Charleston, West Virginia 



Barbara Shay 

New York Department of Education 
Albany, New York 

Kevin Smith 

Literacy Volunteers of New York 
Buffalo, New York 

Richard Stiles 

California Department of Education 
Sacramento, California 

Charles Trott 

Northern Illinois University 
DeKalb, Illinois 

Peter Waite 
Laubach Literacy 
Syracuse, New York 

Sharlene Walker 
Oregon Community College 
Salem, Oregon 
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Local Programs in the Pilot Test of the Follow-up Survey 



Alabama New York 

Randall C. White Rick Stem 

Northeast Alabama Area AE Program Monroe #1 BOCES 

DeKalb County Board of Education 41 O’Connor Road 

DeKalb County Vocational Technical School Fairport, NY 14450 



P. O. Box 765 
Rainsville, AL 
256-638-2957 

Arizona 

Mary Sweeny 

Gary Tang Adult Education Center 
7301 N. 58 th Avenue 
Glendale, A Z 85301 
602-842-8356 (Sweeny) 
602-842-8164 (Office) 

602- 915-4081 (Fax) 

California 

Judy McCoy 

Overfelt Adult Education Center 
1901 Cunningham Avenue 
San Jose, CA 95122 
408-251-2923 
408-251-7572 (Fax) 



716-383-2265 

716- 383-6411 (Fax) 

Ohio 

Robbie Thomas 
Cincinnati Public Schools 
Queen City Vocational Center 
425 Ezzard Charles Drive 
Cincinnati, OH 45203 
513-977-8065 
513-977-8070 (Fax) 

Pennsylvania 

Joyce Kerrick 
Lackawana Junior College 
201 Main Street 
Towanda, PA 18848 

717- 265-3449 
717-265-0876 (Fax) 



Iowa 

Mary Teague 

Eastern Iowa Community College District 

Career Assistance Center 

627 West Second Street 

Davenport, Iowa 52801 

319-326-5319 

319-326-6039 (Fax) 



Tennessee 

Bill Walker 

Knox County Schools 

Adult Basic Education Program 

Knoxville, Tennessee 37917 

423-594-3620 

423-594-1198 (Fax) 
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Idaho 

Shirley Spencer 
Department of Education 
P.O. Box 83720 
Boise, ID 83720 
(202) 332-6931/6933 
(202) 334-4664 (fax) 



States in the State Pilot Test 



Maryland 

Patricia Bennett 
Maryland State Department 
of Education 

200 West Baltimore Street 
Baltimore, MD 21201 
(410) 767-0160 
(410) 333-2099 (fax) 



Rhode Island 

Robert Mason 
Department of Education 
255 Westminster Street 
Shephards Building 
Providence, RI 02903 
(401) 222-4600 x 2180 
(401) 222-2537 (fax) 
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National Reporting System 

Conducted by: 

Pelavin Research Center 
American Institutes for Research 
Washington, D.C. 

Contract U VN970 12001 

Project Staff 

Larry Condelli, Project Director 
Vince Padilla 
January Angeles 
Rob Leland 
Renee Sherman 
Mark Kutner 

John Tibbetts (Consultant) 

Allison Farrar 



U.S. Department of Education 
Office of Vocational and Adult Education 

Trish McNeil, Assistant Secretary for 
Vocational and Adult Education 

Ronald Pugsley, Director 
Division of Adult Education and Literacy 

Mike Dean, Program Specialist 
Division of Adult Education and Literacy 



For: 
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Sample Local Follow-up Survey for Core Measures 




Hello. My name is . I work for . We’re calling people who have recently attended classes at 

our adult education program to find out what happens t them after they leave us. We want to know how you liked the classes 
you took and how adult education classes have affected you, your family, and your job. 

It should take no longer than 10 minutes to answer my questions. Do you have time now for me to ask these questions? 
(Reassure the respondent that any information given to us wilt be strictly confidential). 

First, I’d like to make sure I have the correct information about the class you took. 

A-1. I understand that you were In (TEACHER’S NAME)’s class at (LOCATION. Is that correct? 

□ Yes 

□ No [Obtain correct information] 

A-2. Did you attend teacher’s class until it ended or did you leave before it ended? 

□ Completed [Proceed to Question B-1] 

□ Left before it ended [Proceed to Question A-3] 

A-3. During what month did you stop attending the class or program? 

Month 




B-1. Since the end of your class or program, have you enrolled in any other educational or training programs? 

□ Yes 

□ No [Proc eed to Question C-1] 

B-2. Where are you enrolled? 

□ Other (Specify) 

B-3. In what type of class or classes are you now enrolled? [Do not read choices. Check all that apply.] 

□ English Language Skills 

□ GED/High School 

□ Vocational/Job Training 

□ Community College/College Level 

□ Citizenship 

□ Family literacy 

□ Other (Specify) 

□ DK/Refused 



BEST COPY AVAILABLE 
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C * SeCOMOARV ^CREPB/[|AL 




Did you receive any diplomas, certificates, or degrees at the end of your class or since you left (TEACHER’S) 
class, such as the GED? 



□ Yes 

□ No [Proceed to Question D-1] 

□ DK/Refused [Proceed to Question D-1] 



What type of diploma/certificate/degree did you receive? [Do not read choices to respondent. Check all that 
apply.] 

□ GED 

□ High School Diploma 

□ Certificate of Competence 

□ Associate’s Degree 

□ Bachelor’s Degree 

□ Other 

□ DK/Refused 




D-1. When you first enrolled in the class or program were you: [Read choices.] 



□ Employed at a paying job [Proceed to Question D-4] 

□ Not employed at a paying job and looking for a job [Proceed to Question D-2] 

□ Not employed and not looking for a job [Proceed to E-1] 

□ DK/Refused [End interview] 

D-2. While you were taking (TEACHER’S)’s class did you get a paying job? 

□ Yes 

If yes: What was the name of your employer? [Proceed to Question D-4] 

□ No 

D-3. Since you stopped taking the class, have you gotten a paying job? 

□ Yes 

If yes: What is the name of your employer? ___ 

When did you first get a job after leaving the program? 

□ No [Proceed to Question E - 1] 

D-4. Do you still have that job or do you now have a different job? 

□ Still have same job 

□ Have different job 

What is the name of your current employer? 

□ Lost job, unemployed 

□ DK/Refused 
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«:|||Hi« ulch taking tunc to answer my questions. Your answers wiii be yen: 
;||||||^||e: formation yon gave me will be used to help make adult education programs better and 
hke you who bate; attended; or w child like to atte nd such a . programs !!ii!!lllll 

E-1. Is there anything that I didn’t ask about that you’d like to say? 
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Calling Log 



Interviewer: 



Date & Time 


Name 


Contact (who, nature of conversation, 
any messages left, etc.) 


Status 

(Interview completed, 
scheduled recall) 
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Sample Follow-up Survey 
for Core, Secondary and Other Measures 



BEST COPY AVAILABLE 
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Sample Follow-up Survey for Core, Secondary, and Other Measures 

Hello. My name is . I work for . We're calling people who have recently attended our classes 

at our adult education programs to find out what happens to them after they leave us. We also want to know how you liked the 
classes you attended and how adult education classes have affected you, your family, and your job. 

It should take no longer than 15 minutes to answer my questions. Do you have time now for me to ask these questions? 
(Reassure the respondent that any information given to us wiii be strictly confidential). 




A-1. I understand that you were in (TEACHER’S NAME)’s class at (LOCATION). Is that correct? 

□ Yes 

□ No [ Obtain correct in formation] 



A-2. During what month and year did you enroll in this program? 
Month Year 



A-3. Did you attend the class/program until it ended? 

□ Yes [Proceed to question B-1] 

□ No [Proceed to question A-4] 

A-4. During what month did you stop attending the class or program? 
Month 



A-5. 



What was the main reason you stopped attending the class or program? [Do notread choices to respondent 
Check category that is most closely related to response.] 



□ Achieved reason for enrollment 

□ Completed class 

□ Illness/Incapacity 

□ Lack of child care 

□ Lack of transportation 

□ Family problems 

□ Time or location of services not feasible 

□ Lack of interest 

□ Instruction not helpful 



□ Instructor was not good 

□ Program didn’t satisfy personal goals 

□ Not satisfied with program 

□ Moved 

□ Entered employment 

□ Entered other education or training program 

□ Other (Specify: ) 

□ DK/Refused 




B-1. Did you receive any diplomas, certificates, or degrees since you took this class, such as a GED? 

□ Yes [Proceed to question B-2] 

□ No [Proceed to question C- 1 ] 

□ DK/Refused [Proceed to question C-1] 
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B-2. What type of diploma/certificate/degree did you receive? [Do not read choices to respondent Check all that 
apply] 

□ GED 

□ High School Diploma 

□ Certificate of Competence 

□ Associate’s Degree 

□ Bachelor’s Degree 

□ Other 

□ DK/Refused 
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C-1. Since you stopped attending the class or program, have you enrolled in any other educational or training 
programs? 

□ Yes 

□ No [Proceed to question D-1] 



C-2. Where are you enrolled? 
□ Other (Specify) 



C-3. In what type of class or classes are you now enrolled? [Do not read choices. Check all that apply.] 

□ English Language Skills 

□ GED/High School 

□ Vocational/ Job Training 

□ Community College/College Level 

□ Citizenship 

□ Family literacy 

□ Other (Specify). 

□ DK/Refused 




D-1. While you were enrolled in the class or program, were you receiving any type of public assistance, such as 
food stamps or welfare benefits? 

□ Yes 

□ No [Proceed to question D-3] 

□ DK/Refused [Proceed to question D-3] 



D-2. Are you currently receiving this type of public assistance? 

□ Yes 

□ No 

□ DK/Refused 



D-3. When you first enrolled in the class or program, were you: [Read choices.] 

□ Employed at a paying job [ Proceed to question D-6] 

□ Not employed at a paying job and looking for a job [Proceed to question D-4] 

□ Not employed and not looking for a job [Proceed to question E-1] 

□ DK/Refused [Proceed to question E-1] 
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D-4. While you were taking this class, did you get a paying job? 

□ Yes 

If yes: What was the name of your employer? [Proceed to Question D-6] 

□ No [Proceed to question D-5] 

D-5. Since you stopped taking this class, have you gotten a paying job? 

□ Yes 

If yes: What is the name of your employer? [ Proceed to Question D-6] 

When did you first get a job after leaving the program? 

□ No [Proceed to E-1] 

D-6. Do you still have the same job, have a different job, or have no current job? 

□ Still have the same job 

□ Have a different job 

What is the name of your current employer? 

□ Have no job, unemployed 

□ DK/Refused 
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E-1. Compared to before you attend the class, have you increased your attendance or activities in any of the 
following: [Read choices. Check alt that apply.] 

□ Neighborhood meetings 

□ Meetings of political groups 

□ Volunteer work or meetings for community organizations 

(List: __) 

□ Do not got to meetings or volunteer 

□ DK/Refused 



E-2. Did you register to vote or vote for the first time since you attended the class? 

□ Yes 

□ No 

□ DK/Refused 
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F-1. Do you live with children who are 12 years old or younger? 

□ Yes 

□ No [Proceed to question G-1] 
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F-2. Since you attended the class, how much do you read with your children compared to before you attended the 
class? Do you: 

□ ' Read with children about the same as before 

□ Read with children more than before 

□ Read with children less than before 

□ Do not read with children at all 

□ DK/Refused 

F-3. How often do you visit the library with your child/children now compared to before attending the program? Do 
you: 

□ Go more often 

□ Go the same amount 

□ Go less often 

□ Not go at all 

□ DK/Refusal 

F-4. Is/are the child/children in your home attending school? 

□ Yes [Proceed to question F-5] 

□ No [Proceed to question G-1] 

F-5. Compared to before you attended the class, how much time do you spend helping the school-aged children in 
your home with homework? Do you: 

□ Help about the same 

□ Help more than before 

□ Help less than before 

□ Not help at all 

□ DK/Refused 

F-6. Compared to before you attended the class, how many of your children’s school activities including 
parent/teacher conferences and school assemblies have you gone to? 

□ Attend about the same 

□ Attend more activities 

□ Attend few activities 

□ Do not attend activities 

□ DK/Refused 
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G-1. What is your general opinion of the quality of the class you attended? Is it unacceptable, not very good, 
satisfactory, or excellent? 



□ Unacceptable 

□ Not very good 

□ Good 

□ Excellent 

□ DK/Refused 



G-2. Did (TEACHER’S NAME)’s class meet the expectations you had for it before you enrolled in it? 

□ Yes 

□ No 

□ DK/Refused 
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G-3. Are you not at all likely, somewhat likely, or extremely likely to attend another class or program offered by 
(PROGRAM/CLASS ORGANIZER }1 

□ Not at all likely 

□ Somewhat likely 

□ Extremely likely 

□ DK/Refused 

G-4. What did you like about this class or program? [List all responses.] 



G-5. What did you not like about this class or program? [List all responses.] 




H-1. Is there anything that I didn’t ask about that you’d like to say? 



Guidelines for Conducting the Follow-up Survey 



Appendix B — B1 1 




136 



Measures and Methods for the National Reporting System for Adult Education 



Model Procedures for Conducting the 
Local Follow-up Survey 

This section describes model procedures for conducting a telephone survey designed to 
collect the NRS follow-up measures. The model is offered as guidance to states in designing and 
conducting die follow-up survey. These procedures are not required [ however , and states may 
develop their own procedures for conducting the survey , as long as they meet the NRS requirements 
described in this document. 

The crucial activities to a conducting a telephone survey that produces valid data are to: 

1. Draw a sample of students that reflects the students who attend your program with one or 
more of the four core outcomes as main or secondary goals; 

2. Reach the students sampled and obtain the information from a large majority of them so 
as not to invalidate the sample; and 

3. Train telephone interviewers so that all interviewers ask the survey questions correctly 
and reliably. 

Selecting the Sample 

The procedures below present a method for randomly selecting a sample of 300 students who 
left the program. 

1. Generate a list from your database of names, with telephone numbers and contact 
information, of students who have left the program, organized by the four core follow-up 
measures. Use students' stated main or secondary goals to identify the groups. You 
should also have the exit quarter for students with employment goals. You may use 
separate lists for each of the four follow-up measures or a single list with all students. 

2. Go through the list to identify any individuals who do not have a telephone number or 
any contact information. Cross these names off your list. 

3. We want to compute a sampling fraction that will give us a 300 student sample. The 
sampling fraction is computed by dividing the total number of students to be sampled by 
the desired sample size. If we have 600 students, the sampling fraction is 600/300 = 2. If 
the sampling fraction is not a whole number, it should be rounded to the nearest whole 
number. 

4. From your list, count down the number of students determined by the sampling fraction 
and include that student in the sample and continue this way throughout the entire list. 

For example, in this example the sampling fraction is 2, so include the second student on 
the list and every other student thereafter. When you are finished, you will have a sample 
of 300 students. This is your primary sample. 

5. Create a backup sample of 50 percent more students than your primary sample. 

Randomly select the backup sample in the same way as the original sample. Compute the 
sampling fraction by dividing the number of students you need by the number of students 
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remaining on the list. Use this number to select every w* student from the list. Make 
sure you have a backup sample sufficient for each of the four core outcome measures. 

The backup sample is used to replace students from the primary sample who cannot be 
reached after four attempts (see below). If there are fewer students remaining on your list 
than you need for the backup sample after you have selected your primary sample, all of 
these students will be included in the backup sample. 



Survey Procedures 

Once you have your sample, you can begin calling students and administering the survey. 
Call each person on your primary sample list. If you cannot reach a person despite your best efforts, 
replace that student with a student from the backup sample. 

As you conduct the survey, it is very important to the integrity of the data collected to know 
how many people in the sample were not reached, how many refused to participate, and what the 
reasons for refusal were. For this reason, maintain a calling log during the conduct of the survey. 
Entries in the log should contain the date and time of each call, the name of the caller, and 
information about the call, including: the name of the respondent, whether the person was reached, 
messages left, whether the interview occurred, and explanations for why it did not. The logs should 
be checked daily to identify respondents who need to be re-called. They should also be checked 
against the list of learners in the sample to make sure all members of the sample are being called and 
contacted. Callers should promptly make a log entry for every telephone call they make, whether or 
not the adult learner was reached. This Appendix includes a sample calling log. 

The validity of the survey depends on reaching all or at least a majority of the students in the 
sample. There will be many difficulties, however, in reaching all of the students in the sample. The 
following section describes some of the most common difficulties in reaching people for a telephone 
survey and guidance on how to resolve these problems. 



Problems Reaching Learners on the Telephone 

In most data collection activities, there are predictable kinds of problems that may be 
encountered. Interviewers may be unable to reach the correct person, the learner may not want to 
speak to the caller, or they may have a protective family. Additionally, learners may not want to 
answer some or all survey items; they may be hostile, confused, or just harried. Further, callers may 
be required to answer questions that they are not equipped to answer. 

Interviewers should have a resource person available who can assist with difficult interviews 
or respondents, and complicated questions. This person should have thorough familiarity with the 
NRS and the procedures used to conduct telephone interviews. She or he should monitor interviewer 
telephone logs, provide general oversight during the interviewing process, and could also be 
responsible for the training. 

Accommodation for other languages. Since the sample may include ESL students and 
other non-native English speakers, interviewers are likely to encounter a language barrier in the 
course of data collection. Every effort must be taken to collect information from all non-English 
speakers included in the sample. Accomplishing this may require the program to translate the survey 
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and use interviewers who are fluent in the languages that may be encountered during the interviews. 
The NRS has Spanish and Vietnamese versions of the model survey, available on request. 

When the student cannot be reached immediately. A gatekeeper is a person or situation 
that stands between you and the person with whom you need to talk. Common gatekeepers are 
family members, and even answering machines. 

❖ Reaching a family member or other person 

S Leave a message. The message should be as follows. 

D Interviewer’s name 
D Calling from (name of program) 

D Calling in reference to the adult education program the person attended 
□ Interviewer will call back at another time. 

S Ask a few questions: 

D When is the learner expected back? 

D What and when is the best way to reach her/him? 

S Wait for no more than two days between callbacks. 

S If multiple messages (more than 3-4) have been left, but the learner has not been 
contacted, the learner should be officially listed as a non-respondent on the calling 
log sheet and replaced from the backup sample. 

❖ Reaching voice mail or an answering machine 

S Leave a message. The message should be as follows: 

D Interviewer name and where interviewer is calling from (name of program) 

D Calling in reference to the adult education program the person attended 
D Interviewer will call back at another time 
S Wait for no more than two days between callbacks. 

^ If multiple messages (more than 3-4) have been left, but the learner has not been 
contacted, the learner should be officially listed as a non-respondent on the calling 
log sheet and replaced from the backup sample. 

❖ Reaching a non-working number or a number that just rings 

S Non-working number should be noted on the calling log sheet as not working. 
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S If the number just rings, the day and time the interviewer called should be noted on 
the log sheet, and the learner should be called at a different time. If multiple calls 
(more than 3-4) are made at different times of the day, and there is still no answer, the 
learner should be officially listed as a non-respondent on the calling log sheet and 
replaced from the backup sample. 

Dealing with refusals. The goal of telephone interviews is to obtain information from all the 
people contacted. However, some interviewees may be initially reluctant to participate in the survey. 
The interviewer should try to “convert” refusals whenever possible; callers should, however, never 
become belligerent or upset or insist that a person complete the survey. 

The best way to handle a refusal is for the caller to present himself or herself as confident and 
proud of the work they are doing. The interviewer should indicate that this survey is an important 
way of providing information to the State Department of Education and the adult education program, 
and decisions about adult education will be made based on this information. 

There are several points in the interview when callers may encounter refusals or reluctance. 
The following examples provide ways to handle this. 

Initial refusal. When learners are first reached, they may not be prepared to speak with the 
interviewer. They may be very busy. If this is the case: 

• Ask about the timing: I’m sorry we reached you at a bad time. When might be a more 
convenient time to reach you? Possible solutions include offering to call them a week 
later, a month later, etc., as long as this is recorded so that the follow-up call is made. 

• When the learner has been reached, but absolutely refuses to participate, a complete 
description should be recorded on the calling log and given to the resource person for 
further attempts. 

Confusion-based refusal. Adult learners who are contacted may be confused or wary about 
how the information collected in the interview will be used. For this reason, they may refuse to take 
part in the interview. 

• If the learner wants to know why the survey is being conducted, the interviewer should 
explain the purpose of the study, emphasizing that the information collected has 
important implications for the national adult education program, as well as for the 
program she or he attended. 

• If the learner wants to know how their information will be used, the interviewer should 
assure the learner that the data will be compiled to find out how well adult education 
programs are performing throughout the country and to improve program services. 

Further, all of the answers that the learner gives will be kept confidential and that no 
names or other identifying information will be associated with their answers. Learners 
should also be assured that they were chosen randomly from the pool of adult learners in 
the state. 



B-15 



Appendix B: Sample Surveys 



140 



Measures and Methods for the National Reporting System for Adult Education 



Time- or burden-based refusal. This type of refusal can occur early in the interview, or at a 
later point. Interviewees may be pressed for time and may try to terminate the interview. If this is 
the case: 

• The interviewer should point out that the survey will only take 10-15 minutes, 
acknowledge that the learner’s time is really important, and tell them that their responses 
to the survey questions would be really helpful: I understand that your time is important. 
We really appreciate your input on this issue. It is important to get the perspective of 
adult education students. 

• The interviewer should tell them about the sampling process: Of the [number] students 
that attended the adult education program, you have been selected as one of only 
[number] to represent the program. Your help is important to us. 

If the respondent is still reluctant, one other strategy may be helpful: 

• The interviewer should try to arrange an alternate time: Might there be a better or more 
convenient time to contact you? 

If none of these strategies is successful, the interviewer should NOT try to persuade the 
learner further. The learner should be thanked for their patience, and told that the caller appreciates 
all the demands on their time. The interviewer should then record a complete description on the 
calling-log contact sheet and the student should be replaced from the backup sample. 



Training 

Staff members who will be conducting the telephone interviews should be trained to ensure 
the integrity of the data collected. To collect valid and reliable data, interviewers must be thoroughly 
familiar with both the process of interviewing and the materials to be used for collecting data. The 
actual training, therefore, can be characterized as having two components: the process of conducting 
telephone interviews and the purpose and structure of the NRS. This section provides suggestions on 
appropriate training activities. 



Focus of Training 

Regardless of the survey, any errors, biases, or inconsistencies on the part of the interviewer 
result in some degree of survey error. It should be a goal to minimize this error. Trained 
interviewers are much more likely to accomplish this goal. The desired result is high quality data, so 
that data are comparable from one interview to another, as well as from one state to another. The 
following guidelines should help minimize survey error, and should thus be conveyed to the 
interviewers during their training. 

1. The interviewing process should be standardized. To ensure that this occurs, interviewers 
must read the questions exactly as written and follow the instructions on the survey 
instrument. 
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2. Interviewers should avoid biasing answers by not showing criticism, surprise, approval, 
disapproval, and/or annoyance at a response; recording answers promptly and accurately; 
and probing for clarification when necessary. 

3. Interviews should be completed in the time promised to the respondents. The interview 
is designed to take about 10 minutes. 

4. Interviewers must be familiar with the material, including the meaning of individual 
questions and the definitions of words and phrases contained in the survey instrument. 

5. Administrative issues should be attended to as soon as possible, including making a 
record of EVERY call made, even if the interviewers reached a wrong number, if nobody 
answered, or if a message was left. 

6. Interviewers should have a thorough understanding of the purpose and structure of the 
NRS and the pilot, as explained in this manual. 



Conducting the Training 

Training interviewers can take many forms, including workshops and meetings. There are, 
however, a few techniques which will make the training more meaningful, and thus make the data 
collected more useful and comparable between states. Among these techniques are: 

• Going over the protocol question-by-question. This will give interviewers a familiarity 
with the questions and answers they are likely to get during their telephone calls. It will also 
allow them to become comfortable with the decisions that must be made as the interview 
begins. 

• Conducting mock interviews with adult education office staff or teachers. This simulates 
real world conditions, giving interviewers valuable practice on how to conduct interviews. It 
also allows adult education office staff to identify issues that were not made clear earlier in 
the training process and to identify problems with the data collection procedures in place. 

• Conducting a mini-pilot test with students not included in the official NRS sample in the 
state. This activity will identify previously unconsidered issues and provide the most 
realistic training for the interviewers. It is an excellent last step prior to officially collecting 
data. 



B-17 



Appendix B: Sample Surveys 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTICE 

Reproduction Basis 




This document is covered by a signed "Reproduction Release 
(Blanket)" form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a "Specific Document" Release form. 




This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release form 
(either "Specific Document" or "Blanket"). 




EFF-089 (3/2000) 




